search

CartoDB / bigmetadata

BSD 3-Clause "New" or "Revised" License
43 stars 11 forks source link

Fix acs 2010 2014 #611

Closed javitonino closed 5 years ago

javitonino commented 5 years ago

Point to 2015 geographies to keep retrocompatibility and save disk space, even if it is wrong to do so.

javitonino commented 5 years ago

What made you dismiss that option?

In the current dump (April 2018), we have the 2010 and 2014 data bounded to 2015 geometries. This patch keeps this behaviour even if its incorrect. Main reason is that the 2014 and 2010 tiger is not at the source we use currently for tiger 2015/2016 (some S3 bucket), so including the geometries means developer effort as well as loading time and bigger dumps. Since it's something that has been broken forever, I don't think it's worth it to fix it now.

juanignaciosl commented 5 years ago

In the current dump (April 2018), we have the 2010 and 2014 data bounded to 2015 geometries. This patch keeps this behaviour even if its incorrect

~Fair enough :+1:~

~:rocket:~

juanignaciosl commented 5 years ago

Tests are failing, and it seems to be related:

tasks.meta: INFO: rollback tasks.us.zillow.Zillow_Zip_1_2011_7d584ba935: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (psycopg2.IntegrityError) duplicate key value violates unique constraint "obs_column_table_pkey"
DETAIL:  Key (column_id, table_id)=(us.census.tiger.zcta5_2016_geoidsl, us.zillow.zillow_zip_1_2011_7d584ba935) already exists.
 [SQL: 'INSERT INTO observatory.obs_column_table (column_id, table_id, colname) VALUES (%(column_id)s, %(table_id)s, %(colname)s)'] [parameters: ({'colname': 'region_name_sl', 'table_id': 'us.zillow.zillow_zip_1_2011_7d584ba935', 'column_id': 'us.census.tiger.zcta5_2016_geoidsl'}, {'colname': 'region_name_sc', 'table_id': 'us.zillow.zillow_zip_1_2011_7d584ba935', 'column_id': 'us.census.tiger.zcta5_2016_geoidsl'}, {'colname': 'allhomesplusmultifamily_zri', 'table_id': 'us.zillow.zillow_zip_1_2011_7d584ba935', 'column_id': 'us.zillow.AllHomesPlusMultifamily_Zri'}, {'colname': 'singlefamilyresidencerental_zri', 'table_id': 'us.zillow.zillow_zip_1_2011_7d584ba935', 'column_id': 'us.zillow.SingleFamilyResidenceRental_Zri'}, {'colname': 'singlefamilyresidence_zhvi', 'table_id': 'us.zillow.zillow_zip_1_2011_7d584ba935', 'column_id': 'us.zillow.SingleFamilyResidence_Zhvi'}, {'colname': 'allhomes_zhvi', 'table_id': 'us.zillow.zillow_zip_1_2011_7d584ba935', 'column_id': 'us.zillow.AllHomes_Zhvi'}, {'colname': 'allhomes_medianvaluepersqft', 'table_id': 'us.zillow.zillow_zip_1_2011_7d584ba935', 'column_id': 'us.zillow.AllHomes_MedianValuePerSqft'}, {'colname': 'allhomes_medianrentalpricepersqft', 'table_id': 'us.zillow.zillow_zip_1_2011_7d584ba935', 'column_id': 'us.zillow.AllHomes_MedianRentalPricePerSqft'})]
javitonino commented 5 years ago

Those tests are also broken in master and I've got no idea why. I'll try to see what's happening before merging this.