search

jcaesar / ftp2mfs

Syncing FTP/HTTP/RSYNC into IPFS
10 stars 2 forks source link

ftp2mfs modifies data too much on target MFS #22

Open kallisti5 opened 11 months ago

kallisti5 commented 11 months ago

Just noticed that ftp2mfs modifies the data too much.

config:

source:  rsync://stuff
target: /stuff

source:

   /stuff/things
   /stuff/thingsb
   /stuff/thingsc

destination:

   /stuff/data/things
   /stuff/data/thingsb
   /stuff/data/thingsc
   /stuff/lastsync
   /stuff/mirror
   /stuff/state

Wouldn't it make more sense to maintain the file structure and add a hidden state directory?

For example: destination:

  /stuff/.ftp2mfs/lastsync
  /stuff/.ftp2mfs/mirror
  /stuff/.ftp2mfs/state
  /stuff/things
  /stuff/thingsb
  /stuff/thingsc
jcaesar commented 11 months ago

I originally considered this. Here's why I didn't do that:
The way I used ftp2mfs was to sync, and then do ipfs files stat --hash /stuff/data and publishing that to ipfs name and some dnslinks. With what you suggest, you'd do ipfs files stat --hash /stuff, but then the folder you publish would contain some weird extra files. I wouldn't like that.

You could of course sync to /.stuff-sync and then always copy /.stuff-sync/data to /stuff after syncing.

If you want to try patching that, does just making this bunch of functions https://github.com/jcaesar/ftp2mfs/blob/d7b9c4a3691437902f974f2fb448ecd1dbb06bac/src/tomfs.rs#L36-L68 return the set of values you want work correctly? (There is some copy/modify/replace-original magic going on, so it might not.)

kallisti5 commented 10 months ago

Ah yeah, so my issue is i'm syncing several things from various locations into one set of directories i'm publishing to ipns.

Publish / -> ipns dnslink