SImplify the config table relay

cagov / caldata-mdsa-caltrans-pems

CalData's MDSA project with Caltrans on Performance Measurement System (PeMS) data

https://cagov.github.io/caldata-mdsa-caltrans-pems/

MIT License

5 stars 0 forks source link

SImplify the config table relay #327

Closed pingpingxiu-DOT-ca-gov closed 1 month ago

pingpingxiu-DOT-ca-gov commented 1 month ago

Talked with Ian, we need to bypass kafka for config tables relay.
First option is, a simple cron job that runs 4 times per day on the machine of Oracle Puller.
Pingping tried and failed. The challenge is, that machine is old and installing Snowsql faced difficulties.
Then using the second option, using shared network-drive in lieu of Kafka for data transfer between Oracle Puller and Uploader.
Tried and worked with a minor bug on date conversion.

pingpingxiu-DOT-ca-gov commented 1 month ago

This is a issue originated from #323

pingpingxiu-DOT-ca-gov commented 1 month ago

@ian-r-rose I tried yesterday and the tricky thing with using Oracle Puller machine for uploading directly to snowflake is, it is too old, and installation of snowsql, a relatively new software, has difficulties.

Therefore, we need to adopt the two machine approach, and explore the way to avoid duplications. I tried your next option - using network shared drive for data transfer, works. I need your help on the date format conversion.

ian-r-rose commented 1 month ago

Great, thanks for the update @pingpingxiu-DOT-ca-gov! I can meet before 9 or after our sprint check-in to take a look at the date format.

ian-r-rose commented 1 month ago

Hi @pingpingxiu-DOT-ca-gov I see that file copies have resumed, is this resolved on your end?

jkarpen commented 1 month ago

Ian and Pingping confirmed this has been resolved so closing this issue.