search

2DegreesInvesting / tiltDataPipelines

MIT License
0 stars 0 forks source link

Restructure table definitions #133

Closed ysherstyuk closed 4 months ago

ysherstyuk commented 4 months ago

Hi @SKruthoff,

The main goal of this change is restructuring table definition for more efficient processing. For now only restructuring table.py where we specify schema into separate layers.

SKruthoff commented 4 months ago

Are there no more tables in the monitoring schema. I vaguely remember a test table or something but we might have removed that one along the way.

There are indeed test tables. One is currently in the landingzone and one is in the raw layer, corresponding with test names of these tables. I see they are also used in testing, but not in the main environment right? So they don't need to be mentioned separately in the dictionary?

If they are used in testing we also need them in the dictionaries, since we also want to run the tests.

ysherstyuk commented 4 months ago

@SKruthoff I have made adjustments to my PR. All issues mentioned above are fixed. I have merged develop into my branch with all data quality checks changes.