Open Mr0grog opened 3 years ago
This issue has been automatically marked as stale because it has not had recent activity. It will be closed in seven days if no further activity occurs. If it should not be closed, please comment! Thank you for your contributions.
This project has a nearly complete Ruby port of the Internet Archive’s SURT Python package buried in the
app/lib/
directory: https://github.com/edgi-govdata-archiving/web-monitoring-db/blob/3bb7e8a8960af75f7d05be86f58a88d055cfc79e/app/lib/surt.rb#L3-L20I wrote it because we needed URL canonicalization tools, none of the existing Ruby ones I could find quite met our needs perfectly, and having a method that roughly matched the Internet Archive’s was advantageous. Nobody had written a Ruby port of SURT.
Since we have generally been working to break more reusable, abstract pieces out of the web monitoring projects, this is probably a really good candidate for that on the ruby side. It might be nice to extract it and publish it as a Ruby Gem. (Gem name:
SURT
, repo name:edgi-govdata-archiving/ruby-surt
)