Do less `files/mkdir` calls

TheDiscordian / ipfs-sync

A simple daemon which will watch files on your filesystem, mirror them to MFS, automatically update related pins, and update related IPNS keys.

BSD 3-Clause "New" or "Revised" License

95 stars 14 forks source link

Do less `files/mkdir` calls #29

Closed TheDiscordian closed 3 years ago

TheDiscordian commented 3 years ago

Related to #28, currently we call files/mkdir for every single file updated/added. It'd speed up these operations a lot if we only did this once per directory.

For context I just attempted to update 18k files totalling ~1.1GiB, it's taking over 12 hours on a powerful system with an NVME SSD. This is obviously not good enough.

Each call can take 1-3s!!

TheDiscordian commented 3 years ago

Testing a possible patch for this right now. I think my set took 13 or 14 hours to process. I'll see how long this one takes.

TheDiscordian commented 3 years ago

Fixed in 02a37e96cf6e118647b0da899ca7573332f19fb8