Create harvest source preprocess function in harvest.py - Githubissues

GSA / data.gov

Main repository for the data.gov service

https://data.gov

Other

546 stars 87 forks source link

Create harvest source preprocess function in harvest.py #4718

Closed rshewitt closed 1 month ago

rshewitt commented 2 months ago

User Story

In order to process a harvest source, the harvest runner needs a function to fetch harvest source information from the harvest database via a job id

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

[ ] GIVEN harvest.py \ WHEN the HarvestSource is instantiated \ THEN harvest source information will be fetched and assigned as attributes to the instance \

Background

the harvest runner is only being provided a job id of what to process
the runner will need to fetch the necessary harvest source information in order to process the harvest source ( e.g. url, type, organization, etc...)

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

refactor HarvestSource class to only require the job_id
- refactor all HarvestSource instantiations in the repo
create preprocess function to read harvest source record
implement inside __post_init__ function
create new class inheriting from the critical exception class and raise it appropriately during preprocess

rshewitt commented 2 months ago

updated part of the sketch. this ticket involves refactoring the HarvestSource class and all tests referencing it so some additional time will be needed.