search

sciencehistory / chf-sufia

sufia-based hydra app
Other
9 stars 4 forks source link

WIP Download SOLR from a local URL #1163

Open eddierubeiz opened 5 years ago

eddierubeiz commented 5 years ago

Amend solr_wrapper so it can download SOLR from a local URL. The URL should be accessible to us (both locally and on Travis) but not to anyone else.

jrochkind commented 5 years ago

I am pretty sure solr_wrapper already has a feature to download Solr from whatever URL you give it. Although I don't remember how to use it, needs to be investigated.

In our case, we are thinking of one of our S3 buckets. The sticking point may be authentication issues.

jrochkind commented 5 years ago

If the solr wrapper config files are run through ERB, it might be as simple as embedding via ERB some code that generates a signed S3 URL on every boot.

eddierubeiz commented 5 years ago

Yes, solr_wrapper can definitely download the code (and the checksum) from arbitrary URLs.

jrochkind commented 5 years ago

we'd have to consider costs of downloading from S3 every time CI runs. @sanfordd

eddierubeiz commented 5 years ago

If the solr wrapper config files are run through ERB, it might be as simple as embedding via ERB some code that generates a signed S3 URL on every boot.

They're not -- they use yml for all the paths and urls. There's no opportunity to make changes to the urls at download time .... as solr_wrapper currently stands.

jrochkind commented 5 years ago

@eddierubeiz yml files are often run through ERB first. I believe the solr_wrapper ones might be.

So you can put <%= ruby_code %> in the YML, and it is interpreted, and is expected to result in legal YML. So for instance:

  some_key: '<%= SomeClass.class_method_that_outputs_a_string %>'

Let's see... yes, solr_wrapper does that convenient thing, so that should work:

https://github.com/cbeer/solr_wrapper/blob/50cb347eecc1b21466db76b0e71a2c8d69921a8e/lib/solr_wrapper/configuration.rb#L169

Note the ERB.new in there. So if you put the ERB tags <%= %> in there, it is run through ERB first, executing what's inside the tags and replacing them with whatever that code outputs, before giving the result to the YAML parser.

eddierubeiz commented 5 years ago

Cool. I did not realize this was going on. Never seen this pattern before. Very cool.