Occasionally the downloads of certain will be corrupted. Unfortunately, future invocations of write run directory will not fix this problem, since the corrupted data will be stored in the cache. We should verify the correctness of downloaded files before storing them in the cache. I believe that every object in GCS already has a CRC that can be verified.
Occasionally the downloads of certain will be corrupted. Unfortunately, future invocations of write run directory will not fix this problem, since the corrupted data will be stored in the cache. We should verify the correctness of downloaded files before storing them in the cache. I believe that every object in GCS already has a CRC that can be verified.