search

earthcube / scheduler

Scheduling approaches related to gleaner tooling
Apache License 2.0
0 stars 0 forks source link

partitioned runs may be scheduled even if one is already scheduled. #111

Open valentinedwv opened 3 months ago

valentinedwv commented 3 months ago

Testing, having the runs daily, s3 full, so multiple runs scheduled until it was noticed. Ideally, should not happen. Discussion

Screenshot 2024-06-17 at 9 27 31 AM
valentinedwv commented 3 months ago

probably don't use the context time as part of the run key

https://github.com/earthcube/scheduler/blob/8f8100139dea2451885c6cb0bfb894d7eb836d04/dagster/implnets/workflows/ingest/ingest/sensors/gleaner_summon.py#L73

valentinedwv commented 3 months ago

This will continue to happen. If we use the same run_key in the sensor, it only runs once. If we use a cursor-based one, then it will could run more than once. We might call the graph ql api to see what is running... can't find an api call.

valentinedwv commented 3 months ago

Looks like instance.get_runs here is an discussion from dagster github or get_run_records