Help with update package script

[drtinaz]

Greetings Kevin. I have been reading the documentation for the update package script and have a few questions. I will be using windows 10 and following the instructions in the link you provided for installing bash. Do I place the Packageroot and stockfiles directories in the same directory as setuphelper? Or do they go inside the setuphelper directory? Do I download the entire package being updated into the packageroot folder and unpack it there? It doesn't really matter where the setuphelper directory is placed?

[kwindrem]

For upadatePacakge:

the directory for each package need to be in the same parent directory. updatePacakge attempts to set the packageRoot based on where the updatePackage script is located. So one directory up from SetupHelper.

I located them in my GitHub directory so updatePackage updates the local copy of the repo which I can then manipulate with GitHub desktop.

It does not matter where the stock files are located but updatePackage will make the assumption that it's in the same directory as SetupHelper.

so:

packageRoot SetupHelper updatePackage HelperFiles version setup FileSets ... GuiMods AnotherPackage ... StockVenusOsFiles v3.30~13 opt etc var v3.20 v2.73 ... Yes, those package directories need to be the complete package.

I have not run this script on a Windows machine so let me know how it goes.

[drtinaz]

OK so I installed bash and installed the ubuntu app following the instructions. I can open the terminal and navigate files and folders no problem. I created a little script to copy the files that are needed for my packages from a full extracted image to the appropriate folder and file structure in the proper location per the instructions you gave me. No problem there, I can navigate thru the folder structure and verify that the files copied properly. however, when I run the updatePackage I get the following:

updatePackage: line 113: syntax error near unexpected token `elif'

'pdatePackage: line 113: `elif [ ! -e "$stockFiles" ]; then

I'm thinking maybe the Ubuntu terminal thru windows doesn't know how to handle the script properly? What OS do you use for your developement PC? Maybe i'll just end up having to create another partition and install a unix os to handle this script. What do you think? FYI, I have also tried running with sudo but it doesn't seem to be a permissions issue, but rather a "I don't understand what you want me to do" issue in the terminal.

What do you think?

[kwindrem]

I'm doing my development on MacOs.

Maybe there's a cr/lf end of line issue????

Running a *nix in a virtual machine might be the best way to get going.

I have no Windows machine although I do have Windows under Parallels. So if you aren't making progress I could try that.

[drtinaz]

I tried converting from crlf to lf but made no difference. I also downloaded linux mint and booted from a live cd and ran updatePackage from command prompt and got the same result. I dunno what the story is. There seems to be something that linux doesn't like. I can run other bash scripts successfully though.

[kwindrem]

OK, thanks for the input. I'll need to do some work to test this on something other than my MacBook but I wlll once I'm done with getting conflict resolution into SetupHelper.

[drtinaz]

I did finally get it working. (well, kinda). Now I just need to learn how to use it properly. It was a line ending issue. For some reason the first time I converted from CRLF to LF it still didn't work, but I tried it again this morning and now it does run without syntax errors. Now i'm just getting through the learning curve. I keep getting ERROR version file missing from stock files v3.22 - can't continue. I have placed the version file in the folder but keep getting the same error. I'll keep plugging at it.

Edit: I think I see where I went wrong. Stay tuned....

[drtinaz]

Finally success! Thanks so much for your work. Must get frustrating dealing with knotheads like me that don't know what they are doing. Thanks again.

[kwindrem]

I doubt seriously that you are a "knothead", just someone learning something new. Many thanks for testing this on Windows. I think you maybe the first one to use updatePackage. I would really a detailed report on your experience so I can make things better. Especially what you needed to do to the script to get it to run without syntax errors, and what OS you ended up using. I'd like my document to reflect what actually works !!!

The stock files database is intended to be copies of SECTIONS of an unmodified Venus OS image. So the version file is in /opt/victronenergy. You could use a full copy but they are large and since you need one for each supported version storage suffers. I extract all of /opt and all of /etc and /var/www/venus/styling for my uses. Of course you can scale back if you know what files are needed or add to it if there are other files you update.

Thanks again for your efforts. It takes a community!

[drtinaz]

As far as what I did to use the script on windows, I just followed the link you provided for installing bash, and then installed the ubuntu app from the windows app store. To get the script to run without syntax errors, I opened the script in Visual Studio and just clicked on the line ending setting at the bottom of the page, then selected LF, and saved the file. As you stated above, I had put a version file in the fileset folder instead of copying the /opt/victronenergy folder. Learning curve.

Another issue that I have ran into is that the updatePackage script does not seem to create symlinks properly,(I may not be using the correct terminology here...it's the link within a fileset folder that points to a file in another fileset folder) and github desktop fails to push and results in an error because it doesn't know how to handle those files. I have had to delete the symlinks and delete the COMPLETE file from the effected fileset or the package install results in a "no fileset". I'm sure this has to do with differences between mac os and linux. I wouldn't even know where to begin on this issue.

[kwindrem]

The symlinks issue is strange as ln -s ../blah foo is just basic unix.

even .. to indicate parent directory is known to Windows.

On Mac OS, GitHub Desktop pushes the symlinks properly. According to this: https://blogs.windows.com/windowsdeveloper/2016/12/02/symlinks-windows-10/ Windows requires admin privileges to create symlinks. It also may be that unix symlinks aren't compatible with Windows, but a quick manual check would confirm this. It may also be that the windows version of GitHub Desktop doesn't support them at all. A coulple of articles I found may shed some light on things:

https://stackoverflow.com/questions/5917249/git-symbolic-links-in-windows https://superuser.com/questions/1713099/symbolic-link-does-not-work-in-git-over-windows

I did not always include all file sets and did not create symlinks to other versions and let _checkFileSets fill in the missing files when the package setup script runs. But there were unexplained instances where the install failed because the active file didn't match any .orig files in other file sets. My guess was that something modified the active file from its stock contents since reinstalling the firmware almost always fixed the issue.

[drtinaz]

Enabling developer options did the trick. Sym links now being created and pushed properly.

[kwindrem]

Thanks. I'll do an update to the documents based on your findings.

[drtinaz]

Is update package capable of processing multiple Venus os versions at once, or should they be done in succession one at a time? For example when creating a package and wanting to make the new package backward compatible for some previous os versions?

[kwindrem]

updatePackage relies on processing all supported Venus Os Versions at the same time because it needs to identify common file contents between versions to avoid filling in the same package contents in multiple file sets

If you only introduce one Venus OS version to updatePackage, it will flag others as UNUSED_FILE_SET and suggest manual removal.

[drtinaz]

Kevin I have a couple questions about the updatePackage script. 1) For packages that have .patch files, does updatePackage check those patch files against the stock files for fail/success? 2) Let's say for example that a firmware update changes a stock file in a manner that would cause a .patch file to fail (let's say for example that patch looks at lines 1 thru 6 and inserts some added lines between 3 and 4, but the updated firmware added a new entry at line 4 which would be immediately after the added text, causing the patch to fail.) Is there a provision for alternate .patch files? If patch fails then check altpatch etc? I think the odds of something changing at the exact location a patch needs to be applied is probably low, but it does exist.

[kwindrem]

Yes, I run patch tests during the CHECK operation and prior to install. The patch must return success before I apply it. There is no check for content however as I don't have a way to know what should be in the patched file. In your example, the patch would fail and would cause the package to not install.

The patch mechanism is very basic at this point. It's only useable for version independent files and was intended to extend a package's version independence. BUT it won't always work across many Venus OS versions because of the problem you identified. I was just thinking about you alternate patch file idea. It would however require trying each patch file to see if any succeed. That's not an issue but adds time to the CHECK and install.

Ultimately, I am looking to avoid REPLACING the active so that multiple packages can modify the same file and avoid conflicts. A package uninstall gets tricky because the reverse patch would need to restore the file not to the stock content but to the state as modified by another package. And if the reverse patch fails there would be no way to recover except to revert to the stock content and loose the other package's mods.

[drtinaz]

I really don't see this type of situation occuring very often with patch files. Would it be feasible to have something similar to the version specific FileSets, but rather than a FileSet that is restrictive to one version, have a range of versions? For example if there is a patch file that applies to versions 2.92 thru 3.20, and then a different patch for 3.21 to current and beyond. Maybe could be done with a secondary filelist that points to .patch1 for versions x thru y, and .patch2 for version z and newer.

I'm just throwing some ideas out there. This is way outside my wheelhouse so please forgive me if I'm stepping on your toes or being too intrusive.

[kwindrem]

Supporting a range of firmware versions is what I do with version-dependent replacements.

Replacements one file set whose originals match the original in another file set use a symbolic link for the replacement rather than a second identical copy. Patches could work the same way.

A version dependent file set could have a .patch file rather than a replacement with the actual name. That is rather than PageSettings.qml, the file set could include PageSettings.qml.patch.That part is fairly easy. The difficulty comes in applying the patch to the existing active file and insuring the content is correct: the patch inserted/removed the code properly and in the right place. Hard replacements fail of the active file doesn't match the original (or more recently the modifying package isn't the same as the one attempting the replacement. For patches however, we want the ability for one package to modify a file and allow another package to further modify it. the active file can be replaced with patched result if the patch indicates it succeeds. But if it fails, the package install must fail and any previous patches reversed.

Additionally, the first package to modify a file must also be able to uninstall itself using a reverse patch or the changes provided by the second package could be lost or corrupted.

While this all sounds extremely complicate (and it is) I do think it's the direction things must take as more authors are modifying Venus OS.

gui-v2 adds yet another challenge at least currently. All changes to the WASM image used for remote GUIs must be combined then the image rcompiled. Compilcation is not something that's possible on the GX device.!!!

I do appreciate the discussion and another mind working on the problem. Hopefully our combined efforts can produce something usable.

[drtinaz]

Can you confirm something for me Kevin? If i understand correctly, currently if a single .patch file cannot succeed in patching all of the desired versions of the original file, then the only option is to use version specific filesets with replacement files? There is not currently an option for an alternate .patch file? Or a version specific folder with just an optional .patch file in it?

It's interesting we recently discussed this issue, as I have come across a situation where an additional change was necessary to a patch file which rendered it incompatible with some older os versions. I may have to go back and create some filesets.

[kwindrem]

Currently, yes that is all true.

I am working on a rewrite to the patching mechanism to allow multiple patch files. I'm hoping to reach the point where no file replacements are done. That is, everything is modified using patch.

I started with allowing .patch1, .patch2, etc and found in some cases I needed three or more different patch files to handle firmware back to v2.70. This approach meant that each version of patch file for each modified file would need to be tested which slows the install process Although if I have enough patch file versions to cover all supported firmware, that time could offset the time it takes to create a missing file set.

What I'm playing with now is creating patch files in the version folders. This would accommodate as many patch files as are needed. I will check to see if other patch files would work before creating new ones and right now I'm creating links to the patch file that works for each firmware version.

I'm working toward using patch for modifications to existing files. Version independent files have no originals so will probably need a replacement unless patch can create the desired content starting with an empty file.

Not sure where this will lead. Stay tuned. And I'd appreciate your comments.

[kwindrem]

Update: after doing some testing, I've discovered that 2-3 patches are typically required, however there are some files that need 6-9 different patch files (v2.17 through current betas). This was a test using GuiMods. (Some of these patch files are due to minor differences in the edited replacements (a blank line here and there, different indent, etc.)

Locating patch files in a set of version files like I do with replacements and originals helps in managing a large package such as GuiMods. BUT I realized that using a more generic .patch1, .patch2 ... would also support different platforms like we just discovered for RemoteGPIO and Cerbo/PI. Storing patch files in a version directories minimizes time installing a package since with generic .patch1 ... each patch file would need to be tested to find one that does not fail.

There is the possibility a patch could succeed but produce the incorrect result. In my tests, I identified a few patches that succeeded but produced a different result than the hand edited replacement file! This is less likely with version specific patch files but could still be a problem when patching a file modified by another package, or for a firmware version that is not included in the package.

I have also not identified a way to handle "alternate originals". These are used when the firmware version affects the content of the added file. An alternate original provides onthing for patch to use as an original file since it can be a completely unrelated file.

Patch errors may produce a system that crashes or operates in an unexpected way and regression testing would be difficult since it would depend on a customer's specific package set. As you've seen with gpio_list, it is difficult to explain the problem to customers so whatever mechanism gets implemented, it needs to be rock solid.

Still thinking about the concept though.

Another approach would be to provide different replacement files based on other package installations. This requires collaboration between authors but may be a more reliable mechanism than trying to patch over a previous install.

[kwindrem]

I'm working on a few baby steps that can be done easily to enhance the patching mechanism. Anything more would most likely require a major rewrite.

Allow at least one alternate patch file. A patch will be attempted for all potential patch files and the first one to succeed will be used.

I'm thinking about adding a suffix to .patch files in PatchSources: ...patch-1, ...patch-2, etc .patch would also be checked. suffix could be anything: ...patch-rpi1, ...patch-v3.30, etc. The order that patches are attempted would not be deterministic.

A flag file in PatchSource will enable "patch in place". That is patching the active file based on the current content of the active file, not replacing it with a patched based on the stock file content.

Checks would be made prior to patching in place and the install would abort if a patch error would corrupt the active file. PackageManager would report the patch failure as a conflict.

updatePackage would attempt to create only the ...patch file (no suffix) but would check any existing ...patch-xxx files also.

Comments.