search

ualbertalib / jupiter

Jupiter is a University of Alberta Libraries-based initiative to create a sustainable and extensible digital asset management system. This is phase 2 (Digitization).
https://era.library.ualberta.ca/
MIT License
23 stars 10 forks source link

WIP Add reports generation as ActiveJob #3620

Open lagoan opened 3 weeks ago

lagoan commented 3 weeks ago

Context

We are generating reports for the entities inside era and we have been doing this with external scripts. We are interested in automating them.

Having the report generation inside the code base will also detect changes that break these steps.

What's New

At this point I am looking for feedback on weather to break up the report generation into different jobs, keep as is in a single large job, or other options.

pgwillia commented 3 weeks ago

At this point I am looking for feedback on weather to break up the report generation into different jobs, keep as is in a single large job, or other options.

I think the answer depends on how we see it being used. If we anticipate the need to only run the whole set, this approach makes sense. If we anticipate rerunning specific reports, then breaking the reports into different jobs should be the priority. Testing and error handling/recovery might be simpler in separate jobs.

I think we don't overthink or prematurely optimize it too much at this point. I'd recommend continuing to convert the scripts to a single job. We can always iterate if we find it's not working well based on lived experience. If it works or we only use it once, then we've saved ourselves a bunch of time.