Storage of recorded data from sailing

[takluyver]

We should think about how and where we store the data files we record while running the robot (Rosbags, GPS traces, parameter dumps). I think I have a copy of all of it on my laptop, and there are copies of parts of it stored on Tabarly (group laptop) and Samepage.

So far, we have just under 1GB of data in nearly 300 files. It compresses quite well: a zip or tarball of the directory comes to ~170MB. Obviously this will grow as we do more tests. The largest individual file is ~70MB uncompressed.

• At a minimum, I should try to copy all the data files onto Tabarly so we have a second copy if one laptop gets broken/stolen, and the data is accessible when I'm not there. But Tabarly often comes on field tests with us, so it's not an ideal backup.
• Git & Github don't work well with large files or binary data (Rosbags are both large and binary). In a pinch, we could put the smaller parameter dumps and GPS traces there.
• Samepage has 5GB of storage on the free plan, which should be enough for quite a while, but I think we have to click around in the web interface to upload files. Something that could be scripted & automated would be preferable.
• Google Drive is similar, assuming that there's still no native app for Linux.
• Dropbox is convenient but the free space is only ~2GB, so we'd be using a significant chunk of that unless we zipped up data (which makes it less convenient).
• I might look at setting up Syncthing for all of us who're interested to sync the data folder.

[takluyver]

We are now using Syncthing to copy the data between several computers, including my laptop, my desktop at work, Sophia's laptop and Tabarly. We may go one step further and set up Syncthing on the Raspi itself, but in the meantime it's not hard to pull files off with sftp.

Sophia would like to make the files publicly available as well; we may do this by syncing them into Tony's Dropbox space, as he has extra space.

[tsaoyu]

Sure, I will setup Syncthing first on my desktop. And then write a tiny batch script which sync file between Syncthing folder and Dropbox folder.