Run one-off full load of publication data in prod (for all time)

[peetucket]

TO-DO:

• [x] Confirm production URL of entity resolver
• [x] Ensure production resolver is healthy
• [x] Merge shared_configs change
• [x] Redeploy ETL to prod, to pick up the above change
• [x] Confirm ETL is using prod URL for resolver
• [X] Delete the grants named graph in prod canonical store (and documented here)
• [x] Kick off grants ETL again in a screen session on rialto-etl-prod
• [x] Fix bug with publications CLI (done)
• [ ] Re-deploy ETL with above fix after grants ETL has finished, which unblocks the next step
• [ ] Run publications ETL (exe/publications load -d data/raw/pubs -o data/pubs to harvest full data set):

  /opt/app/rialto/releases/20181130175223/lib/rialto/etl/extractors/web_of_science.rb:17:in `initialize': wrong number of arguments (given 1, expected 0) (ArgumentError)
      from /opt/app/rialto/releases/20181130175223/lib/rialto/etl/cli/publications.rb:73:in `new'
      from /opt/app/rialto/releases/20181130175223/lib/rialto/etl/cli/publications.rb:73:in `extract'
      from /opt/app/rialto/releases/20181130175223/lib/rialto/etl/cli/publications.rb:46:in `load'
      from /opt/app/rialto/shared/bundle/ruby/2.5.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
      from /opt/app/rialto/shared/bundle/ruby/2.5.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
      from /opt/app/rialto/shared/bundle/ruby/2.5.0/gems/honeybadger-4.1.0/lib/honeybadger/plugins/thor.rb:17:in `invoke_command_with_honeybadger'
      from /opt/app/rialto/shared/bundle/ruby/2.5.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
      from /opt/app/rialto/shared/bundle/ruby/2.5.0/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
      from exe/publications:8:in `<main>'

[mjgiarlo]

Researchers have now been extracted and transformed. They are being loaded now. Once done (in 7-8 hours), grants will be loaded, followed by publications.

[mjgiarlo]

Grants are being ETLed now.

[mjgiarlo]

There are two issues with grant ETL in prod at the moment:

1. The rialto-etl-prod shared_configs branch is pointing at the dev entity resolver (see related PR); and
2. Both dev and prod entity resolver are behaving badly. Connections to dev in the ETL are returning Faraday:ConnectionFailed and connections to prod (in my browser, hitting the healthcheck endpoint) are getting 502 gateway errors or 504 gateway timeouts.

I could use some assistance moving this forward when you are available, @aaron-collier

[mjgiarlo]

Before resuming grants ETL, drop the named graph for grants to ensure we don't have dev data in our prod canonical store. See comment in notes.

[mjgiarlo]

Updated current state and TO-DOs in description of this issue. See :point_up:.

[mjgiarlo]

@peetucket How would you feel about re-assigning this issue to you for now, for the sake of knowledge transfer and modeling our ongoing mode of operation for this project? And I will be happy to assist/pair/etc.