Conversation from MS Teams
@jawrainey: I found a bug in the current pipeline that will need addressed before going live as some data is marked as "is_downloaded=False" but is uploaded anyway
@davidverweij: how about adding validators to the pipeline stages which ensures any next step cannot be true unless the previous one is. This prevents error in our coding where we - currently- could easily set any of these to true and mess around!
@jawrainey : Not sure how that maps to use within the pipeline, but can imagine having such validators being useful on the Record object -- would clean up similar logic in current db./main.py I guess
@davidverweij : Aye - this is at the record object (see link). Can throw (value) error when something/one is trying to set one true if the previous ones are not.
@jawrainey : Sounds good -- am not sure how it maps to the Record level as it depends on when they're (Record collections) are used in db/main.py I guess? e.g., within /dags/btf.py, etc -- I guess being explicit and saying "Record.is_ready_for_upload" but might be trickier when working with lists of records? Either way, having solid validation is needed (laugh)
@davidverweij : It should affect all use of the Record class everywhere if the validator is in place at the Record scheme. I.e.; if some code tries to do record.is_uploaded = True and it doesn't pass validation; it fill error.
Conversation from MS Teams @jawrainey: I found a bug in the current pipeline that will need addressed before going live as some data is marked as "is_downloaded=False" but is uploaded anyway
@davidverweij: how about adding validators to the pipeline stages which ensures any next step cannot be true unless the previous one is. This prevents error in our coding where we - currently- could easily set any of these to true and mess around!
@jawrainey : Not sure how that maps to use within the pipeline, but can imagine having such validators being useful on the Record object -- would clean up similar logic in current db./main.py I guess
@davidverweij : Aye - this is at the record object (see link). Can throw (value) error when something/one is trying to set one true if the previous ones are not.
@jawrainey : Sounds good -- am not sure how it maps to the Record level as it depends on when they're (Record collections) are used in db/main.py I guess? e.g., within /dags/btf.py, etc -- I guess being explicit and saying "Record.is_ready_for_upload" but might be trickier when working with lists of records? Either way, having solid validation is needed (laugh)
@davidverweij : It should affect all use of the Record class everywhere if the validator is in place at the Record scheme. I.e.; if some code tries to do record.is_uploaded = True and it doesn't pass validation; it fill error.