m-lab / etl-schema

All schema and views related to the etl pipeline and public bigquery tables.
Apache License 2.0
7 stars 2 forks source link

wehe.scamper1_hopannotation2 requires date filters #182

Open stephen-soltesz opened 4 months ago

stephen-soltesz commented 4 months ago

Due to a policy/configuration change on raw tables, all queries referencing raw tables require a date filter. As a result some views are now out of order, such as https://github.com/m-lab/etl-schema/blob/main/views/wehe/scamper1_hopannotation2.sql

For example, this query results in an error now where previously it did not:

SELECT * FROM `measurement-lab.wehe.scamper1_hopannotation2` WHERE date = "2024-06-03" LIMIT 1000
Screenshot 2024-06-26 at 12 25 35 PM

A resolution is to reference the id and date column through a USING clause here https://github.com/m-lab/etl-schema/blob/main/views/wehe/scamper1_hopannotation2.sql#L34 (there are currently no known variations using ON)

FYI: @zeinabshmeis in case you are still using this query @robertodauria when you have a working example for MSAK consider adding a link here as an illustrative example.