Access the /data URI of the created artifact to download the remote data
An internal server error 500 is thrown
2022-03-31T14:21:00,684 [https-jsse-nio-8080-exec-5] WARN - Could not connect to data source. [exception=(Cannot buffer entire body for content length: 10485760000)]
java.io.IOException: Cannot buffer entire body for content length: 10485760000
at okhttp3.ResponseBody.bytes(ResponseBody.kt:324) ~[okhttp-4.9.3.jar:?]
Camel Routes
If camel routes are used (https://international-data-spaces-association.github.io/DataspaceConnector/CommunicationGuide/v6/Camel#step-2-create-a-generic-endpoint), a 5GB file is successfully transferred. One may just need to make sure that reverse proxies have a sufficient timeout, when accessing the data via /data URL, due to the connector first downloading the file and then forwarding it to the requestor.
If a 10GB file is used following exception is thrown (but as network monitor shows, download is started and stopped after some seconds):
2022-03-31T15:18:43,161 [https-jsse-nio-8080-exec-2] WARN - Caught an exception during route execution. [error=(RouteError(routeId=e463b5a0-5f28-45a6-9660-56b41813ad97, endpoint=https://speed.hetzner.de/10GB.bin, message=Exception occurred during execution on the exchange: Exchange[], timestamp=2022-03-31T15:18:43.161703663))]
2022-03-31T15:18:43,162 [https-jsse-nio-8080-exec-2] WARN - Could not connect to data source. [exception=(Failed to retrieve data.Exception occurred during execution on the exchange: Exchange[])]
io.dataspaceconnector.common.exception.DataRetrievalException: Failed to retrieve data.Exception occurred during execution on the exchange: Exchange[]
at io.dataspaceconnector.common.routing.RouteDataRetriever.get(RouteDataRetriever.java:86) ~[classes/:7.0.2]
Describe the bug There a two ways of fetching data from a REST API. Either one uses Routes, or one uses Artifact.accessUrl.
accessUrl
In latter case, the used library Okhttp v4.9.3 limits the max filesize to Int.MAX_VALUE (https://github.com/square/okhttp/blob/43bc338e8b80625655cd6c0a2ff547da43a14bef/okhttp/src/commonMain/kotlin/okhttp3/internal/-ResponseBodyCommon.kt#L38). Thus, the theoretically max file size is about 2,147 GB
To Reproduce Steps to reproduce the behavior:
Camel Routes
If camel routes are used (https://international-data-spaces-association.github.io/DataspaceConnector/CommunicationGuide/v6/Camel#step-2-create-a-generic-endpoint), a 5GB file is successfully transferred. One may just need to make sure that reverse proxies have a sufficient timeout, when accessing the data via /data URL, due to the connector first downloading the file and then forwarding it to the requestor. If a 10GB file is used following exception is thrown (but as network monitor shows, download is started and stopped after some seconds):