Support non-local file access

Netflix / photon

Photon is a Java implementation of the Interoperable Master Format (IMF) standard. IMF is a SMPTE standard whose core constraints are defined in the specification st2067-2:2013

Apache License 2.0

234 stars 76 forks source link

Support non-local file access #367

Open palemieux opened 3 weeks ago

palemieux commented 3 weeks ago

The current use of the java.io.file API in IMPAnalyzer significantly limits the use of the library. In particular, it requires the IMF to be available on the local filesystem, which is not always possible/practical, e.g., on AWS lambda.

Two options come to mind:

move to the java.nio.file API for which bindings to S3 exist, e.g., https://github.com/awslabs/aws-java-nio-spi-for-s3
add an S3 implementation of ResourceByteRangeProvider and abstract java.io.file operations, e.g., like in https://github.com/Netflix/photon/pull/293

danielhdz13-netflix commented 2 weeks ago

Remote byte access is a high-value add for Photon, and we acknowledge the pain point of java.io.file not just in IMPAnalyzer, but across the library. Netflix is interested in solving this problem by adding remote-file access capabilities.

Regarding the existing S3 support pull request, we have identified several improvements that would be needed before merging. We will add a more detailed review in the PR, but they mostly revolve around backward-compatibility and leaky abstractions. We actually prefer the Java NIO-based approach - this way, Photon can focus on doing what it does best while delegating S3 byte access to another component.