CartoDB / bigmetadata

BSD 3-Clause "New" or "Revised" License
43 stars 11 forks source link

A dependency around `GeomColumns` is wrong #592

Open juanignaciosl opened 6 years ago

juanignaciosl commented 6 years ago

While Tiger was updated and hierarchy was computed, we were hit by this error:

File "/bigmetadata/tasks/base_tasks.py", line 118, in output
    for col_key, col in self.columns().items():
  File "/bigmetadata/tasks/us/census/tiger.py", line 99, in columns
    raise e
  File "/bigmetadata/tasks/us/census/tiger.py", line 87, in columns
    name='Shoreline clipped ' + '_{}'.format(self.year) + col.name,
AttributeError: 'NoneType' object has no attribute 'name'
2018-10-30 10:20:54,349 [INFO]: rollback tasks.us.hierarchy.USLevelInfoFromGeoNames_block_2015_51839c2b6d: 'NoneType' object has no attribute 'name'
2018-10-30 10:20:54,371 [INFO]: Informed scheduler that task   tasks.us.hierarchy.USLevelInfoFromGeoNames_block_2015_51839c2b6d   has status   FAILED

Running USLevelInfoFromGeoNames loads GeoNamesTable, which loads GeonameColumns, which loads ClippedGeomColumns, which fails when OBSColumn is read from a GeomColumns (l. 87). col is Nil.

USLevelInfoFromGeoNames requires GeoNamesTable requires GeonameColumns requires ClippedGeomColumns requires GeomColumns

In that dependency hierarchy, the wrong stuff is at the last one: GeomColumns is not up-to-date. Nevertheless, as GeoNamesTable is "complete", the dependency execution doesn't realize that GeomColumns should be evaluated.

Rerunning GeomColumns manually (make -- docker-run us.census.tiger.GeomColumns --year 2015) fixes the problem, but this should've been handled automatically, as the version was changed.

This is minor, as it has a known workaround.

cc @antoniocarlon @javitonino @ethervoid