bwilab_scripts: divide upload operation into two stages: compression and rsync

[jack-oquin]

I had some trouble Wednesday with some of the uploads. If the compression succeeds but the rsync fails, the log files are left in a state that the upload script no longer handles correctly. It just says there is nothing to upload.

Since the rsync is really the only part requiring bwilab authorization, I'd like to divide the work into separate compression and rsync scripts, perhaps with another convenience script that runs one and then the other. That will make it easier to clean up after a previous incomplete upload. It also significantly reduces the amount of processing done under bwilab authorization.

@pato: comments? (I intend to merge your updates before making this change.)

[jack-oquin]

Also, I want to add an optional argument specifying which directory to search for the logs. With Pato's getopt changes, that will be easier to implement.

The default would be ~/.ros/bwi/bwi_logging for the current user.

[pato]

I like the idea of splitting it up so only the actual uploading is run under bwilab. So we essentially want to separate the script into compress and upload scripts. Where the first takes the prefix and compresses all of the bags matching them (and deletes the originals). And the second takes a prefix and uploads all of the compressed bags matching and uploads them (also deleting local copies). Right?

[jack-oquin]

Yes.

[jack-oquin]

Commit d0baa2c is a first step. It creates a script named bags that does just the compression step. It seems to work, but needs more testing. Then, e206f1b removes the compression logic from the upload script, making it purely an rsync wrapper.

Now, I am thinking about adding an upload (-u) argument to the bags script, so it optionally runs the upload after compressing.

[jack-oquin]

With 56c8675:

• -c compress only (no upload)
• -u upload only (no compress)

Still needs doc updates.

[jack-oquin]

Depending on the file permissions, the bwilab account may not be able to delete the files after sending them.

We could instead delete them under the user's account after the transfer, but that might delete files that were not actually transferred at that time.

[jack-oquin]

@pato suggested moving the kept files to a different subdirectory. Then, deleting with the filename glob in future runs would be safe.

[jack-oquin]

Note that rosbag has a -j parameter to create compressed bags in the first place. We should use that (see: utexas-bwi/bwi_common#46). Once that change has been made, we will no longer need to compress the bags before uploading, and the compressed subdirectory will not be needed.

But, everyone will need to update to that version before we remove the current compression logic.

[jack-oquin]

The -j update has been tested and committed in bwi_logging (utexas-bwi/bwi_common#46). But, we should not depend on that change until other bwi_logging changes are committed and all our users have merged those updates into their workspaces.

I tested that update with the current bags and update scripts and they still work fine.

[jack-oquin]

These commits attempt to fix the -k option by copying the kept files to the kept subdirectory of the original BWI logging folder. It is not fully tested.

[jack-oquin]

Works now for me.