Missing cancelled journeys

samijuhani commented 5 years ago

Example today:

danieldunderfelt commented 5 years ago

These got stuck in the cache for some reason, which resulted in more not loading in. Cache cleared now.

samijuhani commented 5 years ago

Still missing quite alot

danieldunderfelt commented 5 years ago

I'm deploying a fix now that will cache cancellations less, so these should show up soon in all enviuronments (live in dev already).

samijuhani commented 5 years ago

Might need @hellej for this as seems mentioned alerts above are missing from database

samijuhani commented 5 years ago

So looks like we are missing cancelled journeys that have been cancelled after the end time of the journey and partly cancelled journeys.

hellej commented 4 years ago

So looks like we are missing cancelled journeys that have been cancelled after the end time of the journey and partly cancelled journeys.

@samijuhani @danieldunderfelt Is this happening in both dev and prod or only prod?

Currently, more cancellations are saved to dev as also cancellations added to the past are queried from OMM (by HSLdevcom/transitdata-omm-cancellation-source/pull/16). That feature/fix was not yet merged to master since there was a concern (some time ago) that some cancellations may be missing from dev because of it (@samijuhani ?). However, I can confirm that more cancellations have been saved to dev than prod e.g. last_modified > '2019-12-05 00:00:00.000+00' and last_modified < '2019-12-09 00:00:00.000+00' -> 41 in prod vs. 46 in dev.

I can deploy the new cancellation source to stage & prod tomorrow. Will close #486 too

samijuhani commented 4 years ago

Cancelled journeys after journey time has been fixed already in prod also. Only thing to do now with cancelled are partial cancellations

samijuhani commented 4 years ago

Except https://github.com/HSLdevcom/transitlog-ui/issues/486 seems still not to be there in PROD (nor DEV).

samijuhani commented 4 years ago

For example this has been cancelled afterwards and is seen in every env https://reittiloki.hsl.fi/journey/20191211/064500/1092/2/0022_881?date=2019-12-11&mapView=60.21502765949603%2C25.076079368591312&route.routeId=1092&route.direction=2&route.originStopId=1454128&tab=&details-tab=journey-alerts&mapZoom=15&vehicle=0022%2F881&live=false&time=06%3A45%3A00

danieldunderfelt commented 4 years ago

The cancellation in #468 is still missing from envs where it doesn't exist since cancellations (and all events) are only sent once. So if the environment doens't catch it, there's no mechanism to resend it except republishing it or manually entering it into the database. Am I correct, @hellej?

hellej commented 4 years ago

@danieldunderfelt @samijuhani What we know about the cancellation mentioned in #486:

It is in the dev db (@danieldunderfelt confirmed this in the issue)
It is missing from stage & prod that still use the old cancellation source that doesn't query (all) new cancellations added to the past
If it's not shown in dev UI, it may be because citus db is now used there and all cancellation data has not yet been migrated (?) @AkiValiaho
Yes, cancellations (and other events) are only sent once and hence it won't automatically appear in stage & prod even after the new cancellation source is deployed to prod
Old missing cancellations can be queried and inserted to db with some modifications to cancellation source (unique constraints in db will prevent inserting duplicates)

HSLdevcom / transitlog-ui

Missing cancelled journeys #401