best-practice-and-impact / ons-spark

MIT License
9 stars 5 forks source link

Staging tables #70

Closed sam1mitchell1-hub closed 1 year ago

sam1mitchell1-hub commented 1 year ago

Bulk of checkpointing and staging tables code.

Things to note:

For review:

NathanKelly-ONS commented 1 year ago

Just working my way through the R code - one thing I noticed is that the file path set in the config for the animal_rescue csv is set to the animal_rescue.csv in Johanna's user area so I had to change it to read it in - might not be a problem once it's merged as the config might be accurate on the main branch but just wanted to flag just in case.

robertswh commented 1 year ago

Just working my way through the R code - one thing I noticed is that the file path set in the config for the animal_rescue csv is set to the animal_rescue.csv in Johanna's user area so I had to change it to read it in - might not be a problem once it's merged as the config might be accurate on the main branch but just wanted to flag just in case.

Well spotted @NathanKelly-ONS. This is fixed in another pull request.