Add ability to disable staleness checks

[wishnick]

Since our data is immutable we'd like to reduce the calls to S3 for files that are already cached to check if they have changed.

I took a small stab at trying to make this change and my best guess was that it would mean getting rid of this call here to the parent file system, which ultimately delegates down to PrestoS3FileSystem here where it calls S3.

But I wasn't 100% sure if this was the right place or if there was a better place.

[abhishekdas99]

We do make S3 calls (HEAD call) to get the filesize and lastmodified time. Lastmodified time is used for invalidation. In you case if the data is immutable, so we don't need to invalidate. But the code you pointed where we are making getFileStatus call, it also provides the file length which is needed when we are reading the file.

For eg, if the filesize is 1000 bytes and the client wants to read from offset 900 bytes and of length 200 bytes, then the file size comes into picture and limits the read till 1000 bytes. So whenever we open an input stream, we get the above mentioned information from the remote call. And its not possible to completely get rid off the call.

One possible solution we can get is pass this remote call to bookkeeper server. In that case we can cache the file size as well and we can also add a flag to enable invalidation check. Caching the filesize and turning off file invalidation flag will not make getFileStatus call everytime when we open an input steam object. In that case we need to add a new method in thrift layer.

cc @vrajat

[shubhamtagra]

There is a branch to cache file status and listings in rubix: https://github.com/stagraqubole/rubix/tree/cacheMD

That could also help this use case by only reading this information (once per node in worst case) and caching it for subsequent reads. We can work to bring in that change if it helps @wishnick @abhishekdas99