MobilityData / gtfs-validator

Canonical GTFS Validator project for schedule (static) files.
https://gtfs-validator.mobilitydata.org/
Apache License 2.0
290 stars 101 forks source link

ci: aling matrix dimensions to max concurrent jobs #1904

Closed davidgamez closed 1 month ago

davidgamez commented 1 month ago

Summary:

Currently, the number of concurrent jobs is limited by the type of account the organization is holding. As a Team account, we have 60 maximum number of jobs that can concurrently be executed at a time. Having 256 rows in the acceptance tests has two main issues. The first only 60 can be executed concurrently so having more than 60 will make the agents wait and don't add any value as a concurrent job. Secondly, the billable time is computed per second, but the minimum billable time is one minute; all the jobs completed before one minute will be charged one minute, loosing few seconds per small job.

Expected behavior:

Acceptance tests passes and the resources are more efficient used.

Please make sure these boxes are checked before submitting your pull request - thanks!

github-actions[bot] commented 1 month ago

📝 Acceptance Test Report

📋 Summary

✅ The rule acceptance has passed for commit ba5f1073eed683bfe7e85bfcd459945d55c0dc10 Download the full acceptance test report here (report will disappear after 90 days).

📊 Notices Comparison

New Errors (0 out of 1602 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Errors (0 out of 1602 datasets, ~0%) ✅

No changes were detected due to the code change.

New Warnings (0 out of 1602 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Warnings (0 out of 1602 datasets, ~0%) ✅

No changes were detected due to the code change.

🛡️ Corruption Check

0 out of 1602 sources (~0 %) are corrupted.

⏱️ Performance Assessment

📈 Validation Time

Assess the performance in terms of seconds taken for the validation process.

| Time Metric | Dataset ID | Reference (s) | Latest (s) | Difference (s) | |-----------------------------|-------------------|----------------|----------------|----------------| | Average | -- | 3.95 | 3.98 | ⬆️+0.03 | | Median | -- | 1.39 | 1.41 | ⬆️+0.01 | | Standard Deviation | -- | 11.26 | 11.20 | ⬇️-0.06 | | Minimum in References Reports | us-oregon-hut-airport-shuttle-gtfs-635 | 0.52 | 0.56 | ⬆️+0.05 | | Maximum in Reference Reports | gb-unknown-uk-aggregate-feed-gtfs-2014 | 294.60 | 290.62 | ⬇️-3.99 | | Minimum in Latest Reports | us-california-city-of-wasco-gtfs-1788 | 0.53 | 0.52 | ⬇️-0.01 | | Maximum in Latest Reports | gb-unknown-uk-aggregate-feed-gtfs-2014 | 294.60 | 290.62 | ⬇️-3.99 |
📜 Memory Consumption | Metric | Dataset ID | Reference (s) | Latest (s) | Difference (s) | |-----------------------------|-------------------|----------------|----------------|----------------| | Average | -- | 477.07 MiB | 468.55 MiB | ⬇️-8.52 MiB | | Median | -- | 246.09 MiB | 246.50 MiB | ⬆️+425.43 KiB | | Standard Deviation | -- | 875.76 MiB | 801.87 MiB | ⬇️-73.88 MiB | | Minimum in References Reports | us-california-flex-v2-developer-test-feed-1-gtfs-1817 | 34.48 MiB | 34.51 MiB | ⬆️+24.00 KiB | | Maximum in Reference Reports | gb-unknown-uk-aggregate-feed-gtfs-2014 | 10.10 GiB | 10.14 GiB | ⬆️+41.36 MiB | | Minimum in Latest Reports | us-massachusetts-massachusetts-area-express-max-gtfs-431 | 34.49 MiB | 34.48 MiB | ⬇️-8.00 KiB | | Maximum in Latest Reports | gb-unknown-uk-aggregate-feed-gtfs-2014 | 10.10 GiB | 10.14 GiB | ⬆️+41.36 MiB |
github-actions[bot] commented 1 month ago

📝 Acceptance Test Report

📋 Summary

✅ The rule acceptance has passed for commit ba5f1073eed683bfe7e85bfcd459945d55c0dc10 Download the full acceptance test report here (report will disappear after 90 days).

📊 Notices Comparison

New Errors (0 out of 1602 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Errors (0 out of 1602 datasets, ~0%) ✅

No changes were detected due to the code change.

New Warnings (0 out of 1602 datasets, ~0%) ✅

No changes were detected due to the code change.

Dropped Warnings (0 out of 1602 datasets, ~0%) ✅

No changes were detected due to the code change.

🛡️ Corruption Check

0 out of 1602 sources (~0 %) are corrupted.

⏱️ Performance Assessment

📈 Validation Time

Assess the performance in terms of seconds taken for the validation process.

| Time Metric | Dataset ID | Reference (s) | Latest (s) | Difference (s) | |-----------------------------|-------------------|----------------|----------------|----------------| | Average | -- | 3.96 | 4.13 | ⬆️+0.17 | | Median | -- | 1.39 | 1.48 | ⬆️+0.09 | | Standard Deviation | -- | 11.22 | 11.17 | ⬇️-0.05 | | Minimum in References Reports | us-california-city-of-wasco-gtfs-1788 | 0.51 | 0.52 | ⬆️+0.02 | | Maximum in Reference Reports | gb-unknown-uk-aggregate-feed-gtfs-2014 | 292.45 | 288.51 | ⬇️-3.94 | | Minimum in Latest Reports | us-california-city-of-wasco-gtfs-1788 | 0.51 | 0.52 | ⬆️+0.02 | | Maximum in Latest Reports | gb-unknown-uk-aggregate-feed-gtfs-2014 | 292.45 | 288.51 | ⬇️-3.94 |
📜 Memory Consumption | Metric | Dataset ID | Reference (s) | Latest (s) | Difference (s) | |-----------------------------|-------------------|----------------|----------------|----------------| | Average | -- | 481.55 MiB | 472.40 MiB | ⬇️-9.15 MiB | | Median | -- | 246.43 MiB | 246.50 MiB | ⬆️+68.41 KiB | | Standard Deviation | -- | 887.54 MiB | 841.11 MiB | ⬇️-46.43 MiB | | Minimum in References Reports | us-california-flex-v2-developer-test-feed-3-gtfs-1819 | 34.49 MiB | 34.49 MiB | ⬆️+5.84 KiB | | Maximum in Reference Reports | gb-unknown-uk-aggregate-feed-gtfs-2014 | 10.11 GiB | 10.10 GiB | ⬇️-16.72 MiB | | Minimum in Latest Reports | us-florida-citrus-county-transit-gtfs-630 | 34.50 MiB | 34.48 MiB | ⬇️-18.16 KiB | | Maximum in Latest Reports | gb-unknown-uk-aggregate-feed-gtfs-2014 | 10.11 GiB | 10.10 GiB | ⬇️-16.72 MiB |
davidgamez commented 1 month ago

The acceptance test was executed two times, and without many stats, it was completed faster than usual. The time was reduced to ~30 minutes from ~35-45 minutes. Billable time looks down by almost a half. This is not a statistical conclusion; we will monitor the performance of the acceptance tests in subsequent PRs.

https://github.com/MobilityData/gtfs-validator/actions/runs/11465910593