snowplow-incubator / snowplow-lake-loader

Snowplow Lake Loader
Other
0 stars 3 forks source link

Avoid error on duplicate view name #80

Closed istreeter closed 2 months ago

istreeter commented 2 months ago

On rare occasions I have seen errors where Spark complains about creating two temporary tables with the same name. In the loader we create table names based on the window's start time. The error was unexpected because each window should have a different start time.

I believe this is the fix. It ensures view name is computed right at the start of the window, and not after waiting for the table to be initialized. It prevents consecutive windows from picking the same timestamp in the case when the table is very slow to initialize.