spacedriveapp / spacedrive

Spacedrive is an open source cross-platform file explorer, powered by a virtual distributed filesystem written in Rust.
https://spacedrive.com
GNU Affero General Public License v3.0
30.57k stars 896 forks source link

[ENG-349] Archive/Zip #2356

Open brxken128 opened 4 months ago

brxken128 commented 4 months ago

We need a job to archive file(s) and directories. Currently we aren't able to multi-select items within the explorer, so I'll design this job around one object or directory (like the others). There are some thoughts about implementing multi-select support for FS jobs below¹.

We can offer ZSTD and possibly a few other compression algorithms (and levels!), with the zip crate. I have experience with this crate already, and I feel as though it's better than the alternatives. There are a few quirks for Windows machines (mostly \ issues) that I'll have to ensure are thoroughly tested.

Eventually I feel as though we should design our own zip crate - it's not the easiest of tasks but we'd definitely benefit from it in the long-run.

We will also have to implement zip-slip protection for unzipping.

Additionally, we need to ensure that the supported compression algorithms won't cause any build issues, as I believe some of them are as annoying as FFmpeg - I'll need to double check this though.

¹ As a further note, multiple input/source object support probably be achieved by taking a Vec<{path_id}> and iterating through them to create steps for the job. We can use this for copy, erase, cut, delete and pretty much all actions due to the job system.

From SyncLinear.com | ENG-349

brxken128 commented 4 months ago

Agreed with @ericson.ds999, we don't need to broke this job to steps, it can be a 1 big bulky zip and unzip job that does it thing through the library. I don't think pausing it or remembering where it is makes sense here too. Let's not make this complicated and keep it as simple as possible. It just needs work.

brxken128 commented 4 months ago

I don't think that this job can be broken in steps at all. Steps are only needed when we can interrupt the job to be resumed later, which isn't the case. But the new task system enables periodically pauses with a mutable state between them. Without the restriction of being needed to be serializable. On a compression task we would be able to pause it in memory, but if the app was closed, it would need to be started from the beginning.

brxken128 commented 4 months ago

Oh sorry I missed that part, thanks @brxken128! Do you know about this @ericson.ds999?

brxken128 commented 4 months ago

and the current job system doesn't allow for a mutable object to be shared between job steps (which is what the zip crate would require)

brxken128 commented 4 months ago

Why is this blocked?

brxken128 commented 4 months ago

Yeah, we still need zip-slip protection as well as special path handling on Windows (most likely), and the current job system doesn't allow for a mutable object to be shared between job steps (which is what the zip crate would require).

brxken128 commented 4 months ago

As mentioned on Slack, plan is to use a already ready zip crate

brxken128 commented 4 months ago

Might not be blocked but not a priority atm.

brxken128 commented 4 months ago

I'm pretty sure this is still not possible given the current state of the job system. I will look further into it, but we need a mutable state which is not serialised. Maybe a serde(skip)will work, but the job will not be resumable at all.

brxken128 commented 4 months ago

I'm currently unable to continue with this as we're not able to mutably share objects between job steps, when the objects in question cannot be serialized/deserialized.