We do an extra listStatus call for every FS.open() call, in CachingInputStream creation, to get the fileSize and lastModifiedDate. This increase the api calls to clouds and can mean additional costs. We can avoid this by serializing this information into filePath when creating the FileStatus at coordinator.
We do an extra listStatus call for every FS.open() call, in CachingInputStream creation, to get the fileSize and lastModifiedDate. This increase the api calls to clouds and can mean additional costs. We can avoid this by serializing this information into filePath when creating the FileStatus at coordinator.