Use tar and ssh to publish package metadata files.

[nuclearsandwich]

The publish-over-ssh plugin adds an extra configuration step that's hard to manage securely as it does not use the Jenkins ssh-agent and credential system, nor does it support openssh formatted or ed22519 keys, unlike the Jenkins private key credential.

Although it requires a bit more finagling ourselves, I think this is an equally valid way to publish this content which only requires one copy of the ssh private key in the ssh-agent and not an out-of-band system.

I chose sftp over rsync in order to batch the transfers of these smaller files without doing wildly long rsync commands.

[nuclearsandwich]

I wouldn't mind seeing cat << EOF > put_metadata to avoid the 9 extra process invocations, but they do no harm as-is.

In point of fact, I think echo is a shell builtin not a process invocation but it was still 9 extra file IO operations rather than one big one. There's so many layers of templating and quoting that I really didn't like the idea of using a heredoc.

[nuclearsandwich]

I had to change tactics to using find instead of globs to aggregate metadata files.

Using the sftp batch as written had two regressions in behavior:

1. globs that did not have matches resulted in errors

2. file paths were not preserved by sftp so every matching file was put directly into the noetic directory rather than preserving its path relative to noetic.

1 is trivial to fix by prefixing the sftp command with - but 2 is substantially more challenging. After attempting a few sftp and rsync changes, the problem came down to the fact that not every glob will always have matching files and when the glob fails to expand, the glob is passed to the command literally. Which meant that an rsync or scp command would get a glob (that is known not to have any valid expansion) and fail because the file with the glob characters in it does not exist.

So instead we change modes entirely and use find to match regular expressions that are equivalent to the glob expressions. It seemed to me that the most convenient way to transmit the found files, preserving relative paths, was with tar. Which is why the files are tar'd and piped to ssh to be untar'd at the correct path.

[ros-discourse]

This pull request has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/migration-of-docs-ros-org-to-new-web-host/37753/7