search

GenericMappingTools / gmtserver-admin

Cache data and script for managing the GMT data server
GNU Lesser General Public License v3.0
7 stars 3 forks source link

Merge the GMT FTP server into Data Server? #83

Open seisman opened 3 years ago

seisman commented 3 years ago

The GMT FTP site (ftp://ftp.soest.hawaii.edu/gmt) is the site for distributing GMT tarballs. Before we migrated to GitHub, it's the only official site for downloading GMT tarballs.

As mentioned in the bus-factors repository, it has a bus factor = 1, as Paul is the only person who can manage files in the FTP server.

Considering that "FTP is almost dead" (https://filecamp.com/blog/top-5-reasons-ftp-dead/, https://www.ghacks.net/2019/08/16/google-chrome-82-wont-support-ftp-anymore/), should we migrate the GMT FTP server to the Data Server? For example, put all tarballs in the releases subdirectory?

Pros:

  1. the Data server uses HTTP or HTTPS, not FTP.
  2. @PaulWessel @joa-quim @leouieda @seisman all can manage the files in the data server, so the bus factor is 4.
  3. The data server already have many mirrors

We can still keep the FTP server for backward compatibility, but we may retire it after 5-10 years.

seisman commented 3 years ago

Ping @PaulWessel @joa-quim @leouieda for comments.

PaulWessel commented 3 years ago

Yes, we can. It adds ~21 Gb so not too much. Keeping the old server still means maintaining it, by me, though. Do the ubuntus of the world need to be told we changed where things are? Can we retire the old server earlier?

joa-quim commented 3 years ago

I'm afraid + 21 Gb will push my quota close to its limits (~100 Gb) for no good reason. Most of the material in the FTP site is repeated stuff or oooold versions. They can be stored in a single site only.

PaulWessel commented 3 years ago

I also see little value in mirroring old versions. Perhaps the ftp cen be the graveyear for old versions and the release directory can hold the current version only.

leouieda commented 3 years ago

@PaulWessel how about archiving the old releases on Zenodo? Then we can leave the FTP site for now but can be retired without loosing those builds.

PaulWessel commented 3 years ago

We could do that. Any complications of going backwards in time when adding to the same "collection"? Usually we add the next version but now we would add old versions?

seisman commented 3 years ago

Please note that old GMT releases may have multiple tarballs, for example, GMT 4.5.6 have five different tarballs

gmt-4.5.6-doc.tar.bz2
gmt-4.5.6-share.tar.bz2
gmt-4.5.6-src.tar.bz2 
gmt-4.5.6-suppl.tar.bz2 
gmt-4.5.6-triangle.tar.bz2
PaulWessel commented 3 years ago

That is true. Might it be acceptable to repackage these in a single archive to avoid this mess? I expect the demand for 4.5.6 is not urgent.

maxrjones commented 3 years ago

The Zenodo option for old releases seems good. I could try one of these old releases to see if Zenodo will accept multiple tarballs for a single release. I prefer that over repackaging old releases, because that seems more accurate to me. The older releases for GMT 4 and 5 do not already have DOI's, correct?

seisman commented 3 years ago

The older releases for GMT 4 and 5 do not already have DOI's, correct?

Yes.

I prefer that over repackaging old releases, because that seems more accurate to me.

It means more work on us (mainly @PaulWessel), and it may also break the old install instructions for these old versions.

PaulWessel commented 3 years ago

Could we just uncompress to *.tar, add the installer script, then zip the 5 tar files into a zip and register that one instead? Upon extraction the user should have all the pieces.

maxrjones commented 3 years ago

Firefox just disabled FTP support: https://www.mozilla.org/en-US/firefox/88.0/releasenotes/?utm_source=firefox-browser&utm_medium=firefox-desktop&utm_campaign=about-dialog. I would prefer to start putting releases on the Data Server.

maxrjones commented 3 years ago

Could we just uncompress to *.tar, add the installer script, then zip the 5 tar files into a zip and register that one instead? Upon extraction the user should have all the pieces.

Seems OK. What is the motivation for repackaging as a single tarball?

PaulWessel commented 3 years ago

Well, it would be a single file to download but it would contain upon expansion 5 tarballs so that the install script still works. if we truly repackaged into a single tar then we would have to waste time rewriting a GMT4 install script and who needs that headache.

maxrjones commented 3 years ago

Well, it would be a single file to download but it would contain upon expansion 5 tarballs so that the install script still works. if we truly repackaged into a single tar then we would have to waste time rewriting a GMT4 install script and who needs that headache.

OK, sounds good. Is the ftp or data server required for some distributions? If not, should we just use Zenodo and GitHub for future releases rather than continuing with 3+ storage locations?

seisman commented 3 years ago

Is the ftp or data server required for some distributions?

The FTP or data server is required for the mirrors.

maxrjones commented 3 years ago

Is the ftp or data server required for some distributions?

The FTP or data server is required for the mirrors.

I understand that the FTP and data server are required for the mirrors. More specifically, do future releases need to be on these mirrors? Is this a requirement for people to be able to download releases in reasonable amounts of time? There is a trade-off between the maintenance efforts + server size (which could dissuade potential mirror institutions) and convenience for people downloading GMT, so I am trying to understand whether the decision has already been made that the improved download speeds is worth the efforts and space required to continue posting new releases on either the FTP or data server in addition to Zenodo/GitHub.