Web validator retries with bad zip file

MobilityData / gtfs-validator

Canonical GTFS Validator project for schedule (static) files.

https://gtfs-validator.mobilitydata.org/

Apache License 2.0

288 stars 101 forks source link

Web validator retries with bad zip file #1651

Closed jcpitre closed 8 months ago

jcpitre commented 10 months ago

Describe the bug

I fed a (really) bad zip file to the Web validator. The file is just a text file for which I changed the extension to .zip. After that I can see in the web validator logs (here You need access) that the validator failed with the data as expected but then reruns the validator multiple times. Which give the same results since it's the same input data.

Steps/Code to Reproduce

See above

Expected Results

Some error message saying the file is no good and not do any retry with the same data.

Actual Results

According to the logs the system is retrying with the same data a bunch of times. See screenshot

Screenshots

Here are the logs with the retries:

Files used

No response

Validator version

4.2.0

Operating system

irrelevant

Java version

No response

Additional notes

No response

jcpitre commented 10 months ago

A note about the logs: Since the web validator can receive many requests at the same time, it's quite difficult in the logs to know which entry goes with which request. The job id can be used for some logs. Sometimes it's the thread name, but these are reused randomly.

We should some common string for all the logs related to a request so the logs can be easily filtered.

davidgamez commented 10 months ago

A note about the logs: Since the web validator can receive many requests at the same time, it's quite difficult in the logs to know which entry goes with which request. The job id can be used for some logs. Sometimes it's the thread name, but these are reused randomly.

We should some common string for all the logs related to a request so the logs can be easily filtered.

We need to use a logger in the Spring web context that has access to the HTTP span and trace-id.

davidgamez commented 8 months ago

Tasks:

[x] - Use a logger in the Spring web context that has access to the HTTP span and trace-id
[ ] - Identify the corrupted zip and show client an error message
[ ] - Modify web validator cloud run retry policy. When the zip file is corrupted, don't retry.
[ ] - Test with valid and invalid zip files and with cold start function.

qcdyx commented 8 months ago

Based on my research, this bug is caused by an invalid condition of a while loop in the gtfs web validator. It keeps waiting until a report.html is generated. I'll share details in standup. @davidgamez

emmambd commented 8 months ago

Scope of this issue is to add the GCP logging - #1688 and #1689 created for the other tasks.