chore: add memory usage records to the JSON report

MobilityData / gtfs-validator

Canonical GTFS Validator project for schedule (static) files.

https://gtfs-validator.mobilitydata.org/

Apache License 2.0

288 stars 101 forks source link

chore: add memory usage records to the JSON report #1857

Closed davidgamez closed 1 month ago

davidgamez commented 1 month ago

Summary: This PR adds the memory usage to the JSON report. Memory usage is collected in critical points and persists in the JSON report. The added snapshot points are:

GtfsFeedLoader.loadTables: This is taken after the validator loads all files.
GtfsFeedLoader.executeMultiFileValidators: This is taken after the validator executed all multi-file validators
org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate: This is taken for the complete load and validation method.
ValidationRunner.run: This is taken for the complete run of the validator, excluding report generation

There are two main patterns on how to take a memory usage snapshot:

MemoryMonitor annotation: This annotation persists the memory usage in the target method. As a limitation, for methods that have concurrent thread executions, the annotation persists in multiple snapshots. This cannot be very clear when analyzing memory usage.
MemoryUsageRegister: using the registry directly give you more flexibility than the annotation and can be used in cases where MemoryMonitor produces multiple entries on concurrent executed methods.

As the code hasn't been merged yet, this PR acceptance test doesn't contains the difference between main and branch in terms of memory. Closes #1842

Expected behavior:

A field is added to the JSON report with name memoryUsageRecords. Example:


"memoryUsageRecords": [
  {
    "key": "GtfsFeedLoader.loadTables",
    "totalMemory": 2361393152,
    "freeMemory": 579244448,
    "maxMemory": 8589934592,
    "diffMemory": 526389552
  },
  ......
],


- A new section is added to the acceptance tests with a summary of the feeds that memory had increased and decreased.

Please make sure these boxes are checked before submitting your pull request - thanks!

- [x] Run the unit tests with `gradle test` to make sure you didn't break anything
- [x] Add or update any needed [documentation](https://github.com/MobilityData/gtfs-validator/tree/master/docs) to the repo 
- [x] Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
- [x] Linked all relevant issues
- [ ] Include screenshot(s) showing how this pull request works and fixes the issue(s)

github-actions[bot] commented 1 month ago

📝 Acceptance Test Report

📋 Summary

✅ The rule acceptance has passed for commit a98ca7dd0c65ab79214087ba431dd4e867ee0a2b Download the full acceptance test report here (report will disappear after 90 days).

📊 Notices Comparison

New Errors (0 out of 1588 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Errors (0 out of 1588 datasets, ~0%) ✅

No changes were detected due to the code change.

New Warnings (0 out of 1588 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Warnings (0 out of 1588 datasets, ~0%) ✅

No changes were detected due to the code change.

🛡️ Corruption Check

0 out of 1588 sources (~0 %) are corrupted.

⏱️ Performance Assessment

📈 Validation Time

Assess the performance in terms of seconds taken for the validation process.

| Time Metric | Dataset ID | Reference (s) | Latest (s) | Difference (s) | |-----------------------------|-------------------|----------------|----------------|----------------| | Average | -- | 4.05 | 4.15 | ⬆️+0.10 | | Median | -- | 1.43 | 1.51 | ⬆️+0.08 | | Standard Deviation | -- | 11.40 | 11.48 | ⬆️+0.09 | | Minimum in References Reports | us-california-city-of-wasco-gtfs-1788 | 0.51 | 0.53 | ⬆️+0.03 | | Maximum in Reference Reports | gb-unknown-uk-aggregate-feed-gtfs-2014 | 291.44 | 293.77 | ⬆️+2.33 | | Minimum in Latest Reports | us-california-city-of-wasco-gtfs-1788 | 0.51 | 0.53 | ⬆️+0.03 | | Maximum in Latest Reports | gb-unknown-uk-aggregate-feed-gtfs-2014 | 291.44 | 293.77 | ⬆️+2.33 |

📜 Memory Consumption

List of 25 datasets(no reference available).

| Dataset ID | Snapshot Key(Used Memory) | Reference | Latest | |-----------------------------|-------------------|----------------|----------------| | gb-unknown-uk-aggregate-feed-gtfs-2014 | | | | | | GtfsFeedLoader.loadTables | N/A | 10.22 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 10.09 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 10.09 GiB | | | ValidationRunner.run | N/A | 10.37 GiB | | de-unknown-basler-verkehrsbetriebe-gtfs-1092 | | | | | | GtfsFeedLoader.loadTables | N/A | 5.81 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 7.38 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 7.38 GiB | | | ValidationRunner.run | N/A | 9.61 GiB | | nl-unknown-allgo-keolis-gtfs-1077 | | | | | | GtfsFeedLoader.loadTables | N/A | 4.69 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 8.52 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 8.52 GiB | | | ValidationRunner.run | N/A | 7.40 GiB | | be-wallonne-tec-gtfs-1868 | | | | | | GtfsFeedLoader.loadTables | N/A | 3.14 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 3.60 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 3.60 GiB | | | ValidationRunner.run | N/A | 7.70 GiB | | de-unknown-wurzburger-verkehrsverbund-gtfs-1090 | | | | | | GtfsFeedLoader.loadTables | N/A | 6.71 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 6.91 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 6.91 GiB | | | ValidationRunner.run | N/A | 7.54 GiB | | be-vlaams-gewest-de-lijn-gtfs-684 | | | | | | GtfsFeedLoader.loadTables | N/A | 4.16 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 6.99 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 6.99 GiB | | | ValidationRunner.run | N/A | 6.52 GiB | | pl-mazowieckie-warszawski-transport-publiczny-ztm-warszawa-gtfs-1008 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.15 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 6.99 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 6.99 GiB | | | ValidationRunner.run | N/A | 4.15 GiB | | au-new-south-wales-train-replacement-bus-operators-gtfs-1322 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.08 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 6.89 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 6.89 GiB | | | ValidationRunner.run | N/A | 2.71 GiB | | hu-unknown-volanbusz-gtfs-1836 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.05 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 6.72 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 6.72 GiB | | | ValidationRunner.run | N/A | 1.94 GiB | | be-unknown-societe-regionale-wallonne-du-transport-gtfs-1212 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.30 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 6.64 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 6.64 GiB | | | ValidationRunner.run | N/A | 3.80 GiB | | lv-riga-valsts-sia-autotransporta-direkcija-atd-gtfs-992 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.20 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 6.19 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 6.19 GiB | | | ValidationRunner.run | N/A | 6.24 GiB | | de-berlin-verkehrsverbund-berlin-brandenburg-gtfs-782 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.86 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 6.20 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 6.20 GiB | | | ValidationRunner.run | N/A | 2.75 GiB | | us-new-jersey-new-jersey-transit-nj-transit-gtfs-508 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.49 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.44 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.44 GiB | | | ValidationRunner.run | N/A | 6.06 GiB | | gb-unknown-transport-for-greater-manchester-arriva-in-the-north-west-gtfs-1103 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.09 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.05 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.05 GiB | | | ValidationRunner.run | N/A | 5.87 GiB | | ar-buenos-aires-colectivos-buenos-aires-gtfs-1220 | | | | | | GtfsFeedLoader.loadTables | N/A | 5.53 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.16 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.16 GiB | | | ValidationRunner.run | N/A | 5.78 GiB | | pt-lisboa-carris-metropolitana-gtfs-2027 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.40 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.59 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.59 GiB | | | ValidationRunner.run | N/A | 4.28 GiB | | il-unknown-ministry-of-transport-and-road-safety-gtfs-1134 | | | | | | GtfsFeedLoader.loadTables | N/A | 3.42 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 3.08 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 3.08 GiB | | | ValidationRunner.run | N/A | 5.56 GiB | | au-western-australia-transcollie-gtfs-1086 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.08 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.82 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.82 GiB | | | ValidationRunner.run | N/A | 5.34 GiB | | ca-ontario-hamilton-street-railway-gtfs-723 | | | | | | GtfsFeedLoader.loadTables | N/A | 3.15 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.72 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.72 GiB | | | ValidationRunner.run | N/A | 5.16 GiB | | ie-unknown-bus-eireann-gtfs-941 | | | | | | GtfsFeedLoader.loadTables | N/A | 726.89 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.14 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.14 GiB | | | ValidationRunner.run | N/A | 936.66 MiB | | us-oregon-trimet-portland-streetcar-gtfs-247 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.10 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 3.85 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 3.85 GiB | | | ValidationRunner.run | N/A | 5.06 GiB | | pl-mazowieckie-warszawski-transport-publiczny-ztm-warszawa-gtfs-2092 | | | | | | GtfsFeedLoader.loadTables | N/A | 810.55 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.99 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.99 GiB | | | ValidationRunner.run | N/A | 1.10 GiB | | us-illinois-pace-bus-gtfs-390 | | | | | | GtfsFeedLoader.loadTables | N/A | 517.07 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.58 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.58 GiB | | | ValidationRunner.run | N/A | 4.90 GiB | | fi-unknown-porvoon-museorautatie-gtfs-1102 | | | | | | GtfsFeedLoader.loadTables | N/A | 337.93 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.70 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.70 GiB | | | ValidationRunner.run | N/A | 4.85 GiB | | in-kerala-kochi-buses-and-boats-gtfs-1835 | | | | | | GtfsFeedLoader.loadTables | N/A | 384.62 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.68 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.68 GiB | | | ValidationRunner.run | N/A | 4.69 GiB |

jcpitre commented 1 month ago

Meta comment here: We should consider having a separate report for information related to running the validator, like this and timings. While they are related to the size of the datasets, they are not related their semantics or syntax, which ultimately is what interests the user. Plus in the long run they will lose their meaning. e.g. ok, it took 10 minutes to run a year ago, but what about today, after a year worth of changes in the code. Same with memory consumption.

github-actions[bot] commented 1 month ago

📝 Acceptance Test Report

📋 Summary

✅ The rule acceptance has passed for commit bc96e2806385266ebc901eb1f29e9ec616490bca Download the full acceptance test report here (report will disappear after 90 days).

📊 Notices Comparison

New Errors (0 out of 1588 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Errors (0 out of 1588 datasets, ~0%) ✅

No changes were detected due to the code change.

New Warnings (0 out of 1588 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Warnings (0 out of 1588 datasets, ~0%) ✅

No changes were detected due to the code change.

🛡️ Corruption Check

0 out of 1588 sources (~0 %) are corrupted.

⏱️ Performance Assessment

📈 Validation Time

Assess the performance in terms of seconds taken for the validation process.

| Time Metric | Dataset ID | Reference (s) | Latest (s) | Difference (s) | |-----------------------------|-------------------|----------------|----------------|----------------| | Average | -- | 4.08 | 4.16 | ⬆️+0.09 | | Median | -- | 1.43 | 1.51 | ⬆️+0.08 | | Standard Deviation | -- | 11.66 | 11.68 | ⬆️+0.01 | | Minimum in References Reports | us-florida-citrus-county-transit-gtfs-630 | 0.52 | 0.64 | ⬆️+0.12 | | Maximum in Reference Reports | gb-unknown-uk-aggregate-feed-gtfs-2014 | 300.44 | 300.72 | ⬆️+0.28 | | Minimum in Latest Reports | ph-unknown-hm-transport-inc-and-robinsons-malls-gtfs-1105 | 0.60 | 0.59 | ⬇️-0.01 | | Maximum in Latest Reports | gb-unknown-uk-aggregate-feed-gtfs-2014 | 300.44 | 300.72 | ⬆️+0.28 |

📜 Memory Consumption

List of 25 datasets(no reference available).

| Dataset ID | Snapshot Key(Used Memory) | Reference | Latest | |-----------------------------|-------------------|----------------|----------------| | gb-unknown-uk-aggregate-feed-gtfs-2014 | | | | | | GtfsFeedLoader.loadTables | N/A | 10.17 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 9.78 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 9.78 GiB | | | ValidationRunner.run | N/A | 10.01 GiB | | de-unknown-basler-verkehrsbetriebe-gtfs-1092 | | | | | | GtfsFeedLoader.loadTables | N/A | 5.90 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 6.70 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 6.70 GiB | | | ValidationRunner.run | N/A | 10.01 GiB | | de-unknown-wurzburger-verkehrsverbund-gtfs-1090 | | | | | | GtfsFeedLoader.loadTables | N/A | 6.55 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 7.61 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 7.61 GiB | | | ValidationRunner.run | N/A | 8.80 GiB | | il-unknown-ministry-of-transport-and-road-safety-gtfs-1134 | | | | | | GtfsFeedLoader.loadTables | N/A | 3.29 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 7.05 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 7.05 GiB | | | ValidationRunner.run | N/A | 3.74 GiB | | pl-mazowieckie-warszawski-transport-publiczny-ztm-warszawa-gtfs-1008 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.06 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.47 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.47 GiB | | | ValidationRunner.run | N/A | 6.82 GiB | | be-flamande-de-lijn-gtfs-1869 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.79 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 6.53 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 6.53 GiB | | | ValidationRunner.run | N/A | 2.96 GiB | | be-vlaams-gewest-de-lijn-gtfs-684 | | | | | | GtfsFeedLoader.loadTables | N/A | 5.01 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 6.25 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 6.25 GiB | | | ValidationRunner.run | N/A | 5.93 GiB | | de-berlin-verkehrsverbund-berlin-brandenburg-gtfs-782 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.14 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.65 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.65 GiB | | | ValidationRunner.run | N/A | 6.17 GiB | | ar-buenos-aires-colectivos-buenos-aires-gtfs-1220 | | | | | | GtfsFeedLoader.loadTables | N/A | 5.78 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.13 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.13 GiB | | | ValidationRunner.run | N/A | 6.07 GiB | | co-distrito-capital-de-bogota-simur-bogota-gtfs-1204 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.32 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 6.00 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 6.00 GiB | | | ValidationRunner.run | N/A | 2.78 GiB | | gb-unknown-transport-for-greater-manchester-arriva-in-the-north-west-gtfs-1103 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.27 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.89 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.89 GiB | | | ValidationRunner.run | N/A | 5.84 GiB | | be-unknown-societe-regionale-wallonne-du-transport-gtfs-1212 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.24 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 2.76 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 2.76 GiB | | | ValidationRunner.run | N/A | 5.83 GiB | | be-wallonne-tec-gtfs-1868 | | | | | | GtfsFeedLoader.loadTables | N/A | 3.55 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.54 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.54 GiB | | | ValidationRunner.run | N/A | 3.65 GiB | | co-distrito-capital-de-bogota-simur-bogota-gtfs-2012 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.21 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.48 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.48 GiB | | | ValidationRunner.run | N/A | 4.33 GiB | | us-illinois-chicago-transit-authority-cta-gtfs-389 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.66 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 3.53 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 3.53 GiB | | | ValidationRunner.run | N/A | 5.36 GiB | | th-krung-thep-maha-nakhon-office-of-transport-and-traffic-policy-and-planning-gtfs-1831 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.16 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.22 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.22 GiB | | | ValidationRunner.run | N/A | 5.29 GiB | | dk-unknown-rejseplanen-gtfs-1292 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.64 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 3.78 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 3.78 GiB | | | ValidationRunner.run | N/A | 5.24 GiB | | de-baden-wurttemberg-ova-aalen-gtfs-772 | | | | | | GtfsFeedLoader.loadTables | N/A | 956.00 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.04 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.04 GiB | | | ValidationRunner.run | N/A | 5.10 GiB | | ie-unknown-bus-eireann-gtfs-941 | | | | | | GtfsFeedLoader.loadTables | N/A | 725.86 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.09 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.09 GiB | | | ValidationRunner.run | N/A | 890.54 MiB | | de-baden-wurttemberg-db-zugbus-regionalverkehr-alb-bodensee-gtfs-773 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.63 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.69 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.69 GiB | | | ValidationRunner.run | N/A | 5.03 GiB | | nl-unknown-allgo-keolis-gtfs-1077 | | | | | | GtfsFeedLoader.loadTables | N/A | 4.45 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.82 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.82 GiB | | | ValidationRunner.run | N/A | 5.01 GiB | | us-illinois-pace-bus-gtfs-390 | | | | | | GtfsFeedLoader.loadTables | N/A | 508.51 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.01 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.01 GiB | | | ValidationRunner.run | N/A | 549.98 MiB | | it-marche-trenitalia-gtfs-1319 | | | | | | GtfsFeedLoader.loadTables | N/A | 480.00 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.83 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.83 GiB | | | ValidationRunner.run | N/A | 4.97 GiB | | ca-british-columbia-translink-vancouver-gtfs-1222 | | | | | | GtfsFeedLoader.loadTables | N/A | 4.94 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 3.31 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 3.31 GiB | | | ValidationRunner.run | N/A | 4.56 GiB | | tr-kocaeli-kocaeli-gtfs-1122 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.36 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 3.79 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 3.79 GiB | | | ValidationRunner.run | N/A | 4.86 GiB |

davidgamez commented 1 month ago

Meta comment here: We should consider having a separate report for information related to running the validator, like this and timings. While they are related to the size of the datasets, they are not related their semantics or syntax, which ultimately is what interests the user. Plus in the long run they will lose their meaning. e.g. ok, it took 10 minutes to run a year ago, but what about today, after a year worth of changes in the code. Same with memory consumption.

This is interesting, it might take more work to pass the information across the pipeline when analyzing the execution of the validation process, but it might be cleaner for a "regular" consumer. As an alternative we can create another section(field) in the JSON file that is not the metadata. We keep pouring info into the feed metadata as we go. Let's have more thoughts on it. cc: @emmambd

github-actions[bot] commented 1 month ago

📝 Acceptance Test Report

📋 Summary

✅ The rule acceptance has passed for commit 87eb6a62f613118d9e05fe683845aed8a2bb92e0 Download the full acceptance test report here (report will disappear after 90 days).

📊 Notices Comparison

New Errors (0 out of 1588 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Errors (0 out of 1588 datasets, ~0%) ✅

No changes were detected due to the code change.

New Warnings (0 out of 1588 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Warnings (0 out of 1588 datasets, ~0%) ✅

No changes were detected due to the code change.

🛡️ Corruption Check

0 out of 1588 sources (~0 %) are corrupted.

⏱️ Performance Assessment

📈 Validation Time

Assess the performance in terms of seconds taken for the validation process.

| Time Metric | Dataset ID | Reference (s) | Latest (s) | Difference (s) | |-----------------------------|-------------------|----------------|----------------|----------------| | Average | -- | 4.08 | 4.13 | ⬆️+0.05 | | Median | -- | 1.43 | 1.49 | ⬆️+0.06 | | Standard Deviation | -- | 11.62 | 11.56 | ⬇️-0.07 | | Minimum in References Reports | us-california-flex-v2-developer-test-feed-1-gtfs-1817 | 0.53 | 0.67 | ⬆️+0.14 | | Maximum in Reference Reports | gb-unknown-uk-aggregate-feed-gtfs-2014 | 296.89 | 293.81 | ⬇️-3.08 | | Minimum in Latest Reports | us-florida-citrus-county-transit-gtfs-630 | 0.59 | 0.57 | ⬇️-0.03 | | Maximum in Latest Reports | gb-unknown-uk-aggregate-feed-gtfs-2014 | 296.89 | 293.81 | ⬇️-3.08 |

📜 Memory Consumption

List of 25 datasets(no reference available).

| Dataset ID | Snapshot Key(Used Memory) | Reference | Latest | |-----------------------------|-------------------|----------------|----------------| | gb-unknown-uk-aggregate-feed-gtfs-2014 | | | | | | GtfsFeedLoader.loadTables | N/A | 9.96 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 10.01 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 10.01 GiB | | | ValidationRunner.run | N/A | 10.35 GiB | | de-unknown-basler-verkehrsbetriebe-gtfs-1092 | | | | | | GtfsFeedLoader.loadTables | N/A | 6.70 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 6.36 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 6.36 GiB | | | ValidationRunner.run | N/A | 10.02 GiB | | be-wallonne-tec-gtfs-1868 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.94 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.31 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.31 GiB | | | ValidationRunner.run | N/A | 8.40 GiB | | il-unknown-ministry-of-transport-and-road-safety-gtfs-1134 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.97 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 7.43 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 7.43 GiB | | | ValidationRunner.run | N/A | 4.09 GiB | | de-unknown-wurzburger-verkehrsverbund-gtfs-1090 | | | | | | GtfsFeedLoader.loadTables | N/A | 6.57 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 7.18 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 7.18 GiB | | | ValidationRunner.run | N/A | 7.00 GiB | | nl-unknown-allgo-keolis-gtfs-1077 | | | | | | GtfsFeedLoader.loadTables | N/A | 4.30 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 6.83 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 6.83 GiB | | | ValidationRunner.run | N/A | 4.75 GiB | | be-vlaams-gewest-de-lijn-gtfs-684 | | | | | | GtfsFeedLoader.loadTables | N/A | 6.68 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 6.70 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 6.70 GiB | | | ValidationRunner.run | N/A | 6.53 GiB | | de-berlin-verkehrsverbund-berlin-brandenburg-gtfs-782 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.22 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.14 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.14 GiB | | | ValidationRunner.run | N/A | 6.66 GiB | | hu-unknown-volanbusz-gtfs-1836 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.20 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.73 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.73 GiB | | | ValidationRunner.run | N/A | 6.55 GiB | | ar-buenos-aires-colectivos-buenos-aires-gtfs-1220 | | | | | | GtfsFeedLoader.loadTables | N/A | 5.97 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.10 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.10 GiB | | | ValidationRunner.run | N/A | 6.31 GiB | | dk-unknown-rejseplanen-gtfs-1292 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.77 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.77 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.77 GiB | | | ValidationRunner.run | N/A | 6.24 GiB | | us-new-jersey-new-jersey-transit-nj-transit-gtfs-508 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.45 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.34 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.34 GiB | | | ValidationRunner.run | N/A | 5.96 GiB | | de-baden-wurttemberg-db-zugbus-regionalverkehr-alb-bodensee-gtfs-773 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.78 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.47 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.47 GiB | | | ValidationRunner.run | N/A | 5.81 GiB | | be-unknown-societe-regionale-wallonne-du-transport-gtfs-1212 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.27 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.70 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.70 GiB | | | ValidationRunner.run | N/A | 2.90 GiB | | ie-unknown-bus-eireann-gtfs-941 | | | | | | GtfsFeedLoader.loadTables | N/A | 728.00 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.10 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.10 GiB | | | ValidationRunner.run | N/A | 5.67 GiB | | ee-unknown-abuss-ou-gtfs-1095 | | | | | | GtfsFeedLoader.loadTables | N/A | 662.67 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.38 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.38 GiB | | | ValidationRunner.run | N/A | 770.55 MiB | | pl-mazowieckie-warszawski-transport-publiczny-ztm-warszawa-gtfs-1008 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.13 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 2.97 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 2.97 GiB | | | ValidationRunner.run | N/A | 5.31 GiB | | pt-lisboa-carris-metropolitana-gtfs-2027 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.87 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 2.67 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 2.67 GiB | | | ValidationRunner.run | N/A | 5.26 GiB | | fr-provence-alpes-cote-dazur-palmbus-gtfs-925 | | | | | | GtfsFeedLoader.loadTables | N/A | 4.47 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.19 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.19 GiB | | | ValidationRunner.run | N/A | 5.25 GiB | | se-unknown-trafiklab-gtfs-1320 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.22 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.42 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.42 GiB | | | ValidationRunner.run | N/A | 5.18 GiB | | de-baden-wurttemberg-bodo-verkehrsverbund-gtfs-769 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.11 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.97 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.97 GiB | | | ValidationRunner.run | N/A | 5.07 GiB | | br-sao-paulo-sao-paulo-transporte-sptrans-gtfs-8 | | | | | | GtfsFeedLoader.loadTables | N/A | 642.60 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.00 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.00 GiB | | | ValidationRunner.run | N/A | 5.05 GiB | | us-colorado-regional-transportation-district-rtd-gtfs-178 | | | | | | GtfsFeedLoader.loadTables | N/A | 527.17 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.78 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.78 GiB | | | ValidationRunner.run | N/A | 5.04 GiB | | pl-pomorskie-zarzad-transportu-miejskiego-gdansk-ztm-gdansk-gtfs-1014 | | | | | | GtfsFeedLoader.loadTables | N/A | 911.23 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.01 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.01 GiB | | | ValidationRunner.run | N/A | 1.09 GiB | | pl-mazowieckie-warszawski-transport-publiczny-ztm-warszawa-gtfs-2092 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.28 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 3.63 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 3.63 GiB | | | ValidationRunner.run | N/A | 4.95 GiB |

github-actions[bot] commented 1 month ago

📝 Acceptance Test Report

📋 Summary

✅ The rule acceptance has passed for commit 3934be2094e9c342b77fd32f569409873f39a1a7 Download the full acceptance test report here (report will disappear after 90 days).

📊 Notices Comparison

New Errors (0 out of 1588 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Errors (0 out of 1588 datasets, ~0%) ✅

No changes were detected due to the code change.

New Warnings (0 out of 1588 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Warnings (0 out of 1588 datasets, ~0%) ✅

No changes were detected due to the code change.

🛡️ Corruption Check

0 out of 1588 sources (~0 %) are corrupted.

⏱️ Performance Assessment

📈 Validation Time

Assess the performance in terms of seconds taken for the validation process.

| Time Metric | Dataset ID | Reference (s) | Latest (s) | Difference (s) | |-----------------------------|-------------------|----------------|----------------|----------------| | Average | -- | 4.03 | 4.13 | ⬆️+0.09 | | Median | -- | 1.40 | 1.46 | ⬆️+0.06 | | Standard Deviation | -- | 11.58 | 11.81 | ⬆️+0.24 | | Minimum in References Reports | us-california-flex-v2-developer-test-feed-3-gtfs-1819 | 0.51 | 0.51 | ⬆️+0.00 | | Maximum in Reference Reports | gb-unknown-uk-aggregate-feed-gtfs-2014 | 300.34 | 309.50 | ⬆️+9.16 | | Minimum in Latest Reports | us-california-flex-v2-developer-test-feed-3-gtfs-1819 | 0.51 | 0.51 | ⬆️+0.00 | | Maximum in Latest Reports | gb-unknown-uk-aggregate-feed-gtfs-2014 | 300.34 | 309.50 | ⬆️+9.16 |

📜 Memory Consumption

List of 25 datasets(no reference available).

| Dataset ID | Snapshot Key(Used Memory) | Reference | Latest | |-----------------------------|-------------------|----------------|----------------| | gb-unknown-uk-aggregate-feed-gtfs-2014 | | | | | | GtfsFeedLoader.loadTables | N/A | 10.05 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 10.17 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 10.17 GiB | | | ValidationRunner.run | N/A | 10.28 GiB | | de-unknown-basler-verkehrsbetriebe-gtfs-1092 | | | | | | GtfsFeedLoader.loadTables | N/A | 5.86 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 6.68 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 6.69 GiB | | | ValidationRunner.run | N/A | 9.61 GiB | | nl-unknown-allgo-keolis-gtfs-1077 | | | | | | GtfsFeedLoader.loadTables | N/A | 4.17 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 7.79 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 7.79 GiB | | | ValidationRunner.run | N/A | 5.33 GiB | | de-unknown-wurzburger-verkehrsverbund-gtfs-1090 | | | | | | GtfsFeedLoader.loadTables | N/A | 6.56 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 7.22 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 7.22 GiB | | | ValidationRunner.run | N/A | 7.74 GiB | | be-vlaams-gewest-de-lijn-gtfs-684 | | | | | | GtfsFeedLoader.loadTables | N/A | 4.40 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 7.63 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 7.63 GiB | | | ValidationRunner.run | N/A | 7.43 GiB | | hu-unknown-volanbusz-gtfs-1836 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.97 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 6.59 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 6.59 GiB | | | ValidationRunner.run | N/A | 7.41 GiB | | be-unknown-societe-regionale-wallonne-du-transport-gtfs-1212 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.42 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 7.17 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 7.17 GiB | | | ValidationRunner.run | N/A | 4.65 GiB | | il-unknown-ministry-of-transport-and-road-safety-gtfs-1134 | | | | | | GtfsFeedLoader.loadTables | N/A | 3.11 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 7.16 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 7.16 GiB | | | ValidationRunner.run | N/A | 3.75 GiB | | au-new-south-wales-train-replacement-bus-operators-gtfs-1322 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.06 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.48 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.48 GiB | | | ValidationRunner.run | N/A | 7.14 GiB | | de-berlin-verkehrsverbund-berlin-brandenburg-gtfs-782 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.13 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.47 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.47 GiB | | | ValidationRunner.run | N/A | 6.99 GiB | | ar-buenos-aires-colectivos-buenos-aires-gtfs-1220 | | | | | | GtfsFeedLoader.loadTables | N/A | 5.71 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.14 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.14 GiB | | | ValidationRunner.run | N/A | 6.29 GiB | | ie-unknown-bus-eireann-gtfs-941 | | | | | | GtfsFeedLoader.loadTables | N/A | 725.92 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.01 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.01 GiB | | | ValidationRunner.run | N/A | 5.58 GiB | | de-hamburg-hamburger-verkehrsverbund-gmbh-hvv-gtfs-1226 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.13 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.83 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.83 GiB | | | ValidationRunner.run | N/A | 5.53 GiB | | au-western-australia-transcollie-gtfs-1086 | | | | | | GtfsFeedLoader.loadTables | N/A | 780.94 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.97 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.97 GiB | | | ValidationRunner.run | N/A | 5.51 GiB | | br-rio-de-janeiro-city-transport-secretary-of-rio-de-janeiro-gtfs-1791 | | | | | | GtfsFeedLoader.loadTables | N/A | 521.00 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.18 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.18 GiB | | | ValidationRunner.run | N/A | 928.61 MiB | | pl-mazowieckie-warszawski-transport-publiczny-ztm-warszawa-gtfs-1008 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.18 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 2.72 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 2.72 GiB | | | ValidationRunner.run | N/A | 5.07 GiB | | be-wallonne-tec-gtfs-1868 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.93 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.06 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.06 GiB | | | ValidationRunner.run | N/A | 3.34 GiB | | de-baden-wurttemberg-sbg-gtfs-918 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.13 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.80 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.80 GiB | | | ValidationRunner.run | N/A | 5.05 GiB | | pl-mazowieckie-warszawski-transport-publiczny-ztm-warszawa-gtfs-2092 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.28 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 3.61 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 3.61 GiB | | | ValidationRunner.run | N/A | 4.93 GiB | | us-illinois-pace-bus-gtfs-390 | | | | | | GtfsFeedLoader.loadTables | N/A | 514.49 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.78 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.78 GiB | | | ValidationRunner.run | N/A | 316.00 MiB | | us-california-los-angeles-county-metropolitan-transportation-authority-gtfs-29 | | | | | | GtfsFeedLoader.loadTables | N/A | 773.65 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.78 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.78 GiB | | | ValidationRunner.run | N/A | 744.58 MiB | | au-queensland-translink-south-east-queensland-translink-seq-gtfs-668 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.23 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.75 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.75 GiB | | | ValidationRunner.run | N/A | 1.25 GiB | | in-kerala-kochi-buses-and-boats-gtfs-1835 | | | | | | GtfsFeedLoader.loadTables | N/A | 374.52 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.72 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.72 GiB | | | ValidationRunner.run | N/A | 4.74 GiB | | us-colorado-regional-transportation-district-gtfs-2054 | | | | | | GtfsFeedLoader.loadTables | N/A | 585.77 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.46 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.46 GiB | | | ValidationRunner.run | N/A | 4.73 GiB | | co-distrito-capital-de-bogota-simur-bogota-gtfs-2012 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.15 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.72 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.72 GiB | | | ValidationRunner.run | N/A | 3.72 GiB |

github-actions[bot] commented 1 month ago

📝 Acceptance Test Report

📋 Summary

✅ The rule acceptance has passed for commit 8a087622d5c6292bf738bfe0a9f315f3c32a7bad Download the full acceptance test report here (report will disappear after 90 days).

📊 Notices Comparison

New Errors (0 out of 1588 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Errors (0 out of 1588 datasets, ~0%) ✅

No changes were detected due to the code change.

New Warnings (0 out of 1588 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Warnings (0 out of 1588 datasets, ~0%) ✅

No changes were detected due to the code change.

🛡️ Corruption Check

0 out of 1588 sources (~0 %) are corrupted.

⏱️ Performance Assessment

📈 Validation Time

Assess the performance in terms of seconds taken for the validation process.

| Time Metric | Dataset ID | Reference (s) | Latest (s) | Difference (s) | |-----------------------------|-------------------|----------------|----------------|----------------| | Average | -- | 4.06 | 4.13 | ⬆️+0.07 | | Median | -- | 1.42 | 1.48 | ⬆️+0.06 | | Standard Deviation | -- | 11.65 | 11.85 | ⬆️+0.20 | | Minimum in References Reports | ph-unknown-hm-transport-inc-and-robinsons-malls-gtfs-1105 | 0.51 | 0.55 | ⬆️+0.04 | | Maximum in Reference Reports | gb-unknown-uk-aggregate-feed-gtfs-2014 | 299.78 | 302.65 | ⬆️+2.87 | | Minimum in Latest Reports | us-california-catalina-express-gtfs-299 | 0.52 | 0.54 | ⬆️+0.02 | | Maximum in Latest Reports | gb-unknown-uk-aggregate-feed-gtfs-2014 | 299.78 | 302.65 | ⬆️+2.87 |

📜 Memory Consumption

List of 25 datasets(no reference available).

| Dataset ID | Snapshot Key(Used Memory) | Reference | Latest | |-----------------------------|-------------------|----------------|----------------| | gb-unknown-uk-aggregate-feed-gtfs-2014 | | | | | | GtfsFeedLoader.loadTables | N/A | 9.92 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 9.81 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 9.81 GiB | | | ValidationRunner.run | N/A | 10.02 GiB | | de-unknown-basler-verkehrsbetriebe-gtfs-1092 | | | | | | GtfsFeedLoader.loadTables | N/A | 5.90 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 6.53 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 6.53 GiB | | | ValidationRunner.run | N/A | 9.94 GiB | | de-unknown-wurzburger-verkehrsverbund-gtfs-1090 | | | | | | GtfsFeedLoader.loadTables | N/A | 6.57 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 7.36 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 7.36 GiB | | | ValidationRunner.run | N/A | 8.98 GiB | | nl-unknown-allgo-keolis-gtfs-1077 | | | | | | GtfsFeedLoader.loadTables | N/A | 4.47 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 7.05 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 7.05 GiB | | | ValidationRunner.run | N/A | 8.04 GiB | | be-vlaams-gewest-de-lijn-gtfs-684 | | | | | | GtfsFeedLoader.loadTables | N/A | 6.30 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 7.75 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 7.75 GiB | | | ValidationRunner.run | N/A | 7.42 GiB | | il-unknown-ministry-of-transport-and-road-safety-gtfs-1134 | | | | | | GtfsFeedLoader.loadTables | N/A | 3.23 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 3.56 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 3.56 GiB | | | ValidationRunner.run | N/A | 6.04 GiB | | de-berlin-verkehrsverbund-berlin-brandenburg-gtfs-782 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.05 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 6.03 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 6.03 GiB | | | ValidationRunner.run | N/A | 2.04 GiB | | pl-mazowieckie-warszawski-transport-publiczny-ztm-warszawa-gtfs-1008 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.09 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 3.56 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 3.56 GiB | | | ValidationRunner.run | N/A | 5.90 GiB | | ar-buenos-aires-colectivos-buenos-aires-gtfs-1220 | | | | | | GtfsFeedLoader.loadTables | N/A | 5.65 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.13 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.13 GiB | | | ValidationRunner.run | N/A | 5.87 GiB | | be-unknown-societe-regionale-wallonne-du-transport-gtfs-1212 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.30 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.68 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.68 GiB | | | ValidationRunner.run | N/A | 3.05 GiB | | be-wallonne-tec-gtfs-1868 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.96 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.55 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.55 GiB | | | ValidationRunner.run | N/A | 3.79 GiB | | dk-unknown-rejseplanen-gtfs-1292 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.70 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.08 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.08 GiB | | | ValidationRunner.run | N/A | 5.55 GiB | | be-flamande-de-lijn-gtfs-1869 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.99 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.47 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.47 GiB | | | ValidationRunner.run | N/A | 1.79 GiB | | us-oregon-trimet-portland-streetcar-gtfs-247 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.12 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.21 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.21 GiB | | | ValidationRunner.run | N/A | 5.42 GiB | | us-illinois-chicago-transit-authority-cta-gtfs-389 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.69 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 3.53 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 3.53 GiB | | | ValidationRunner.run | N/A | 5.35 GiB | | pt-lisboa-carris-metropolitana-gtfs-2027 | | | | | | GtfsFeedLoader.loadTables | N/A | 2.11 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 2.69 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 2.69 GiB | | | ValidationRunner.run | N/A | 5.28 GiB | | fr-provence-alpes-cote-dazur-palmbus-gtfs-925 | | | | | | GtfsFeedLoader.loadTables | N/A | 4.47 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.17 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.17 GiB | | | ValidationRunner.run | N/A | 5.24 GiB | | de-baden-wurttemberg-db-zugbus-regionalverkehr-alb-bodensee-gtfs-773 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.82 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.88 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.88 GiB | | | ValidationRunner.run | N/A | 5.22 GiB | | au-south-australia-adelaide-metro-gtfs-660 | | | | | | GtfsFeedLoader.loadTables | N/A | 655.01 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 5.13 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 5.13 GiB | | | ValidationRunner.run | N/A | 688.38 MiB | | us-colorado-regional-transportation-district-rtd-gtfs-178 | | | | | | GtfsFeedLoader.loadTables | N/A | 524.97 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.77 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.77 GiB | | | ValidationRunner.run | N/A | 5.04 GiB | | us-new-jersey-new-jersey-transit-nj-transit-gtfs-508 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.30 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.31 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.31 GiB | | | ValidationRunner.run | N/A | 4.92 GiB | | de-bayern-bbs-schapfl-gtfs-770 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.46 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.76 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.76 GiB | | | ValidationRunner.run | N/A | 4.91 GiB | | gb-unknown-go-north-east-gtfs-1935 | | | | | | GtfsFeedLoader.loadTables | N/A | 440.30 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.64 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.64 GiB | | | ValidationRunner.run | N/A | 4.88 GiB | | us-california-los-angeles-county-metropolitan-transportation-authority-gtfs-29 | | | | | | GtfsFeedLoader.loadTables | N/A | 771.65 MiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.87 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.87 GiB | | | ValidationRunner.run | N/A | 834.83 MiB | | us-hawaii-thebus-gtfs-10 | | | | | | GtfsFeedLoader.loadTables | N/A | 1.17 GiB | | | GtfsFeedLoader.executeMultiFileValidators | N/A | 4.34 GiB | | | org.mobilitydata.gtfsvalidator.table.GtfsFeedLoader.loadAndValidate | N/A | 4.34 GiB | | | ValidationRunner.run | N/A | 4.85 GiB |

emmambd commented 1 month ago

@davidgamez @jcpitre re: your comments, I like how this currently works which is memoryUsageRecord as a separate section of the JSON report. I agree if we want to extend these kinds of performance checks, moving them to a separate report might make sense (for example, when we provide more detailed and specific insights into parsing failures). However, I think this is sufficient for now.