search

ssl-hep / ServiceX_frontend

Client access library for ServiceX
https://servicex-frontend.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
5 stars 11 forks source link

Fix minio handling of buckets in rare cases #458

Closed ponyisi closed 2 months ago

ponyisi commented 2 months ago

Sometimes listing the bucket for an in-progress transformation gives objects that look like directories, not files. These have no file size reported and would raise an exception due to the coercion in the ResultFile constructor. Avoid these problematic objects.

codecov[bot] commented 2 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 82.02%. Comparing base (2dfb069) to head (49cf33d). Report is 1 commits behind head on 3.0_develop.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## 3.0_develop #458 +/- ## ============================================ Coverage 82.02% 82.02% ============================================ Files 26 26 Lines 1363 1363 ============================================ Hits 1118 1118 Misses 245 245 ``` | [Flag](https://app.codecov.io/gh/ssl-hep/ServiceX_frontend/pull/458/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ssl-hep) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/ssl-hep/ServiceX_frontend/pull/458/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ssl-hep) | `82.02% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ssl-hep#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

ponyisi commented 2 months ago

Yeah - in any case this is strictly better behavior than we had before, because we would have just crashed and lost everything. I'm not 100% what circumstances lead to this but we should in any case validate the full number of expected files in the bucket at the end of transforms.