[FR] Update an archive when a (text) file is modified

[redtide]

This is a feature that also lxqt-archiver misses, so I ask here first because Arqiver is my first choice, though I could ask you also for it in case if could be accepted: I had to install FileRoller for this, sometimes I need to modify manually a built AUR package and it comes handy by dblclicking the file in the archive, it gets opened by default editor, than save the file: when FileRoller gets back the focus asks me if I want to update the archive with the modified file.

[tsujan]

It may be beyond the time I can currently dedicate to this project — it needs file monitoring, lots of new codes and many changes.

That being said, I keep your report open because, first, it's a good idea, and, second, a simple method might come to my mind at some point.

This is a feature that also lxqt-archiver misses

Sadly, after its creator, I've been the only dev who has worked on it extensively (porting Arqiver's features to it). Implementing this feature might be easier with Arqiver because Qt codes are much handier than GLib ones — and much more compact too.

[redtide]

Thank you, just a though on it, if useful: you said about file monitoring, like a watcher? Because I had in mind to just check that file was changed when gain back the focus:

• I dblclick or open with Arqiver on the archive
• Arqiver opens together with the archive
• I dblclick on a text file to open it, Arqiver save the "currentOpenedTextFilePath", loose focus and the file is opened on the default editor
• I save the file in the editor, so the file is changed in /tmp (I guess)
• Arqiver regain focus, looks at the recorded file opened in /tmp and see it changed
• Dialog if to update the archive -> Yes -> recompress the modified temp file

not sure if this was what you had in mind or having some "parallel file watcher object" to manage the entire archive, this looks to me simpler, but had no much experience with file watching code.

[tsujan]

Yes, something like that came to my mind too. It works based on viewing the current item, reactivation of the window, and checking the modification time.

But there are complications. For example, the user may view multiple files by opening them one after the other, and they may be in different folders. Or he may view a file, move it to another virtual desktop, edit it, and close Arqiver's window by middle clicking its task-bar icon, without activating it first; etc.

[redtide]

But there are complications. For example, the user may view multiple files by opening them one after the other, and they may be in different folders.

a path/checksum openedFiles map?

Or he may view a file, move it to another virtual desktop, edit it, and close Arqiver's window by middle clicking its task-bar icon, without activating it first; etc.

OK, this wants a file watcher :smile: or no save for the sake of simplicity/first implementation; the user should know that to save changes would need to reach the app window. The dialog would ask to save showing a listbox|view with the file names.

[tsujan]

I took the easiest approach and tested by adding an "Update Archive" button to the tool-bar and Edit menu, without file monitoring (I might add monitoring or date checking later, but I don't want to complicate the code unnecessarily). The button will show a prompt when the patch is complete (no prompt in my tests though). If the user accepts it, the files will be replaced in the archive.

It worked with all archives that were managed by bsdtar and gzip. Both of them updated the existing files correctly in the archive.

Actually, it can be tested with the current version of Arqiver too. Once you view a file from an archive that's managed by bsdtar (like a zip archive), a temporary folder is created in /tmp. You could click "Add Directory", go into it, and add the top directories inside it to the archive (or files, if the archive doesn't have a directory). Then bsdtar will update the modified files.

But 7z doesn't do that. I might disable "Update Archive" button with it if no easy way is found for it. I found no command-line option for replacing a folder/file inside a 7z archive either.

[tsujan]

A screenshot:

[tsujan]

Update: Added a file watcher. It enables the button as soon as a file is modified. Also, a prompt will be shown on quitting if a file is modified but the archive isn't updated yet.

However, this note in Qt doc worries me a little: "There may be a system dependent limit to the number of files and directories that can be monitored simultaneously...."

[redtide]

Wow thank you to work on it already!

Regarding the Qt documentation note: it seems harmless to me but still annoying, because something more to work on; being aware about this you should also warn the user that some files can't be monitored for changes caused by an unknown reason (not just the limit AFAIU from the sentence above) to avoid them nasty surprises. This at least if you call that function only when the user opens files, otherwise it's impossible to handle and a simple message box can't be used if it could be a multi selection.

[tsujan]

While I thought everything was OK, I encountered an obstacle: The files weren't updated but duplicated in the archive. Arqiver doesn't show duplicated files (in the same folder), but lxqt-archiver does. Strange that bsdtar can create such a semi-corrupt archive.

The GUI is fine. What I need is more basic: a clean command for updating an archive. Do you know of any?

[redtide]

I don't use much the command line except when using CI, but I just notice a -u option on both tar and bsdtar, or I haven't understand the question.

[redtide]

OK it was no, but maybe this can be somewhat helpful? IIUC tar does what we want and on BSD is called gtar.

[tsujan]

The -u option is only for uncompressed tar archives.

From what I've read here and there, it seems that tar (or bsdtar) can't update files in compressed archives; it just can duplicate them, which is bad. If so, the only way is uncompressing, updating and compressing again.

OK it was no, but maybe this can be somewhat helpful?

It's about uncompressed tar.

[redtide]

I gave for grant that that would work in both cases :-P

[tsujan]

My conclusion so far:

If this is just about uncompressing and recompressing, the user can do it himself. File-Roller seems to do the same thing, With a 1-MiB archive, I was able to see that File-Roller first created a new archive and then overwrote the original one.

To avoid uncompressing and recompressing, I even tested by removing the original files first and adding the modified files afterward. It was faster than File-Roller, but the problem was that the parent directories were duplicated, such that, beside each directory that contained a modified file, an empty directory with the same name appeared. Neither Arqiver not File-Roller showed the duplication, but lxqt-archiver did — empty folders were also visible with command-line.

I don't think uncompressing and recompressing is a good approach. It's slow with average archives (a few megabytes) and intolerably slow with heavy ones. It's just good with light archives.

All in all, there's no unsolved problem about the Qt GUI. Only if a magic command is found...

[redtide]

There is no hurry, so do what you think is best and take your time. I have no interest on modify files in big archives, I have no memory of other examples than Arch packages to change few lines to the .BUILDINFO, I just recall that sometimes it happens and such feature comes very handy to me.

[tsujan]

Another update (I write these updates to prevent going in wrong directions again in case of failures):

This may be an illusion, but I think I've found the "magic command". First, the files which are going to be updated should be removed from the archive. Then the modified files should be added to the archive with appropriate parent paths (the -C option of bsdtar):

bsdtar -c -a -f temp_archive_path -C common_parent_path modified_files_paths @original_archive

Apparently, the problem I encountered yesterday was only because of putting wrong values of common_parent_path and modified_files_paths in the above command (inside the code).

I modified the same 1-MiB archive of yesterday, and lxqt-archiver didn't show any duplication. The speed was that of File-Roller. Will do more tests...

[tsujan]

Done in https://github.com/tsujan/Arqiver/commit/5d243a4b9a8e095b7a5cc98044d8d1f349cdc50a — didn't want to delay it while my mind was warm.

It passed my tests, but your tests would be very appreciated.

Thanks for the good idea!

[redtide]

Thank you again! I'll back on this ASAP with some testing.

[redtide]

I haven't made a deep test, just my usual way to do things and it does its job, the "update" button (which reminds me some app, maybe was synaptics on Debian) looks better than a msgbox request dialog. The only thing that I noticed is the scrollbar jump after the update because reloading, but also Fileroller does it, I never noticed because it doesn't use a treelist (if not enabling the sidebar), so the scrollbar is almost never visible there. All fine! 👍🏽

[tsujan]

Thanks for testing it!

The only thing that I noticed is the scrollbar jump after the update

Its cause is outside this change, somewhere else in the code. I don't remember where, but I'll check the code. What I remember is that, for some reason that I don't remember, I preferred not to expand the tree ;)

[redtide]

It would be nice with a left sidebar and view only the selected folder on the right (a la PCManFM), but it's just cause the habit after a long time using fileroller (LXDE gtk2, then I switched to LXQT few years ago). Anyway, Arqiver is good as is.

[tsujan]

I intentionally didn't do that in Arqiver. To me, it's an obstacle. You could consider it as a "character trait" of this app.

As for the unexpanded tree after reloading (your previous comment), I remembered the reason (mostly, for not taking the CPU time unnecessarily). For now, my focus is on the functionality itself, but I'll see if a better behavior on reloading could be implemented without extra CPU usage with heavy archives.

Please don't hesitate to tell me about any issue you may encounter regarding this functionality.

[tsujan]

After investigating the tree expansion, I saw that it was correct and not related to the scroll jump (my previous theory). The jump is random and happens because a file is first removed and then added, and Qt miscalculates the scrollbar position while repopulating the list. A workaround may be be possible though.

While checking the code, I took a look at bsdtar's manual and accidentally encountered an interesting option that made viewing or extracting of single files much faster with heavy archives (e.g., with ~30,000 files). Will add it to Arqiver after more tests.

[redtide]

The jump is random and happens because a file is first removed and then added, and Qt miscalculates the scrollbar position while repopulating the list. A workaround may be be possible though.

I thought was normal after a tree update. about the workaround I don't think it's necessary, though I don't remember if it's possible in Qt to save the bar position to restore after the "update".

While checking the code, I took a look at bsdtar's manual and accidentally encountered an interesting option that made viewing or extracting of single files much faster with heavy archives (e.g., with ~30,000 files). Will add it to Arqiver after more tests.

Good news!

[tsujan]

I thought was normal after a tree update

It seems to me that Qt does it only when some limits are exceeded. Under the usual circumstances, it shouldn't do it. Maybe GTK has a similar behavior.

about the workaround I don't think it's necessary,

I'm not giving it a high priority. I've just encountered several occasions where simple workaround were possible — some of them are already included in the code.

As for high priority, my preliminary tests show that 7z may also be covered, but with a different approach simpler than that of bsdtar.After a change to the code, I updated a few 7z archives correctly, including a password-protected one. More tests are needed...

[tsujan]

Covered 7z and used the fast-read option with bsdtar in https://github.com/tsujan/Arqiver/commit/45fc672c9cfee5f3c76873e799a43602a063bd52. Also, made the update button invisible with read-only archives, and fixed a small issue with gzip.

It seems complete now :) Probable issues will be fixed if found.

[redtide]

Great! Yeah, no need to be perfect from start :)

[tsujan]

Last but not least, this was the best feature request I'd received for a long time. And I enjoyed implementing it.

I'd also be glad if it was possible to port this code to lxqt-archiver. The fact is that it isn't. Arqiver uses bsdtar and 7z to cover a vast variety of archives, while lxqt-archiver relies on different commands, being based on GLib, engrampa and file-roller.

[redtide]

Thank you, I'm glad you like it!

lxqt-archiver relies on different commands, being based on GLib, engrampa and file-roller.

Uhm, I'm a bit confused, this feature came from using file-roller

[tsujan]

this feature came from using file-roller

Good point. If it was there when engrampa was forked from file-roller, then the core functions should be somewhere in its jungle of codes, and we could make use of them in the Qt GUI.

[tsujan]

Oops! I found an old bug that can also affect this. Fortunately, it's rare — I haven't encountered it in the real world, but I can reproduce it with a test archive.

If an archive has two paths this_path and that_path/this_path, where this_path starts from the root of the archive, then if the user removes or edits the file in this_path, the file at that_path/this_path will also be removed.

The fix is as easy as using the regex symbol ^. Will fix it in the next commit, alongside other changes.

[tsujan]

The fix is in https://github.com/tsujan/Arqiver/commit/52cf985c7cd38e9e9e4dec8483e9f8cc3bb40079, alongside other small enhancements.

I might not add comments to this page about probable enhancements/fixes regarding this functionality. It'll be developed indefinitely, like other features.

[redtide]

thank you again, I'm using the AUR -git package so I can keep it updated and have the latest improvements. I wonder why this application is not in the official repositories in Arch.

[tsujan]

You're very welcome.

I wonder why this application is not in the official repositories in Arch.

There's no predictable rule, Sometimes you see an app with a small number of users in the official repositories; sometimes a popular app like Lyx (which I use on a regular basis) isn't in them.

[redtide]

Maybe a ready AUR non-git version might help 😄