Deploy new OBS Dump - Githubissues

talos commented 8 years ago

Located on S3 here: link removed

Adds tons 'o' new fun facts, including but not limited to:

Business/industrial info for Spain
Demographics in England & Wales
DMAs in the US

@rafatower

rafatower commented 8 years ago

Last deploy was 2016_05_23_9270d0701c for staging users (blue/green), which I believe was never put in production.

I've just deployed 2016_06_15_d428377abd and set it up for staging users. Let me know when you're satisfied with it to put it in production as well.

talos commented 8 years ago

Thanks @rafatower ! Sorry it took me some time to get on testing this.

Unfortunately it doesn't appear that jkrauss.cartodb.com, which is a team account and should be on the whitelist for staging, has access to the new data. Can you doublecheck that account is working with the new data?

rafatower commented 8 years ago

do you have any quick way of checking the version that you're running against?

otherwise we should have something like SELECT OBS_db_version() returning 2016_06_15_d428377abd from a simple table within the dump.

rafatower commented 8 years ago

just figured out where it lies in the config. I can confirm that:

the whitelist is empty at present (which I guess it shouldn't), and
the current dump in prod is obs_2016_05_23_9270d0701c

rafatower commented 8 years ago

Config updated adding team org members to whitelist

rafatower commented 8 years ago

Got these errors in our integration tests:

REMOTE ERROR: schema "cdb_observatory" does not exist'

Rolled back, just left your user jkrauss in the whitelist for the moment.

rafatower commented 8 years ago

Probably my fault...

rafatower commented 8 years ago

Way better now. The problem is that I had to grant certain permissions not done by deploy scripts. For the record:

GRANT USAGE ON SCHEMA cdb_observatory TO geocoder_api;
GRANT SELECT ON ALL TABLES IN SCHEMA cdb_observatory TO geocoder_api;
GRANT USAGE ON SCHEMA observatory TO geocoder_api;
GRANT SELECT ON ALL TABLES IN SCHEMA observatory TO geocoder_api;

I left it set up for just your user jkrauss. If everything is ok I can activate to the rest of the team first thing on Monday.

There's one of the integration tests failing:

07:18:08 ======================================================================
07:18:08 FAIL: test_if_obs_search_is_ok (test.integration.test_data_observatory_functions.TestDataObservatoryFunctions)
07:18:08 ----------------------------------------------------------------------
07:18:08 Traceback (most recent call last):
07:18:08   File "/home/ubuntu/workspace/CartoDB-DATASERVICES-API-Integration-testing/test/integration/test_data_observatory_functions.py", line 135, in test_if_obs_search_is_ok
07:18:08     assert_equal(result['id'], 'es.ine.t1_1')
07:18:08 AssertionError: u'us.census.acs.B01003001_quantile' != 'es.ine.t1_1'
07:18:08 -------------------- >> begin captured stdout << ---------------------
07:18:08 Executing query: https://jkrauss.cartodb.com/api/v1/sql?q=SELECT id FROM OBS_Search('total_pop') LIMIT 1;&api_key=*****
07:18:08 
07:18:08 --------------------- >> end captured stdout << ----------------------
07:18:08 -------------------- >> begin captured logging << --------------------
07:18:08 requests.packages.urllib3.connectionpool: INFO: Starting new HTTPS connection (1): jkrauss.cartodb.com
07:18:08 requests.packages.urllib3.connectionpool: DEBUG: "GET /api/v1/sql?q=SELECT%20id%20FROM%20OBS_Search('total_pop')%20LIMIT%201;&api_key=**** HTTP/1.1" 200 None
07:18:08 --------------------- >> end captured logging << ---------------------

(against your user, api_key omitted)

I don't know whether that test makes sense. Please take a look and let me know:

https://github.com/CartoDB/dataservices-api/tree/development/test https://github.com/CartoDB/dataservices-api/blob/b33730aae3b1c2de4917782c014e80de8b189753/test/integration/test_data_observatory_functions.py#L135

Have a nice one!

talos commented 8 years ago

Thanks! Taking a look at that test and doing additional validation here.

talos commented 8 years ago

This is good to deploy to production @rafatower . Tickets referenced above either address or solve test failures, which were not the fault of the meta/data itself.

rafatower commented 8 years ago

For the record: dump 2016_06_15_d428377abd in production now.

Sorry about the delays. Getting more confident and polishing everything :dancer:

talos commented 8 years ago

🔥

CartoDB / observatory-extension

Deploy new OBS Dump #113