mp3-10
commented
5 years ago NextCloud 16 the same problem :(
Open grantemsley opened 5 years ago
mp3-10
commented
5 years ago NextCloud 16 the same problem :(
wayneyaoo
commented
3 years ago Would love to add this support to the external storage app.
Just wanna confirm that the support isn't complete right? Code change being merged but not functional doesn't make sense at all.
jgallucci32
commented
3 years ago Confirmed this is still an issue with Nextcloud 19.0.3. I even spent half the day getting this to work with Azure Stack Hub (on-prem) Blob storage with a custom SSL certificate. Was all excited and got to testing an saw the error and this reported issue. I will keep digging but it definitely seems to be an issue with the Azure storage driver in Nextcloud.
jgallucci32
commented
3 years ago Here is some additional information from the logs to hopefully help identify the issue
[objectstore] Error: MicrosoftAzure\Storage\Common\Exceptions\ServiceException: Fail:
Code: 411
Value: Content-Length HTTP header is missing.
details (if any): <?xml version="1.0" encoding="utf-8"?><Error><Code>MissingContentLengthHeader</Code><Message>Content-Length HTTP header is missing.
RequestId:dc10ea71-aebf-076e-4386-b7cb312a2fea
Time:2021-05-24T04:22:11.3406813Z</Message><HeaderName>Content-Length</HeaderName></Error>. at <<closure>>
0. /var/www/html/3rdparty/microsoft/azure-storage-common/src/Common/Internal/ServiceRestProxy.php line 406
MicrosoftAzure\Storage\Common\Internal\ServiceRestProxy::throwIfError(GuzzleHttp\Psr7\Response {}, 201)
1. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 203
MicrosoftAzure\Storage\Common\Internal\ServiceRestProxy->MicrosoftAzure\Storage\Common\Internal\{closure}("*** sensitive parameters replaced ***")
2. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 174
GuzzleHttp\Promise\Promise::callHandler(2, "*** sensitive parameter replaced ***", [GuzzleHttp\Prom ... }])
3. /var/www/html/3rdparty/guzzlehttp/promises/src/RejectedPromise.php line 40
GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}("*** sensitive parameters replaced ***")
4. /var/www/html/3rdparty/guzzlehttp/promises/src/TaskQueue.php line 47
GuzzleHttp\Promise\RejectedPromise::GuzzleHttp\Promise\{closure}("*** sensitive parameters replaced ***")
5. /var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php line 118
GuzzleHttp\Promise\TaskQueue->run()
6. /var/www/html/3rdparty/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php line 145
GuzzleHttp\Handler\CurlMultiHandler->tick()
7. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 246
GuzzleHttp\Handler\CurlMultiHandler->execute(true)
8. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 223
GuzzleHttp\Promise\Promise->invokeWaitFn()
9. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 267
GuzzleHttp\Promise\Promise->waitIfPending()
10. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 225
GuzzleHttp\Promise\Promise->invokeWaitList()
11. /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php line 62
GuzzleHttp\Promise\Promise->waitIfPending()
12. /var/www/html/3rdparty/microsoft/azure-storage-blob/src/Blob/BlobRestProxy.php line 1809
GuzzleHttp\Promise\Promise->wait()
13. /var/www/html/lib/private/Files/ObjectStore/Azure.php line 109
MicrosoftAzure\Storage\Blob\BlobRestProxy->createBlockBlob("nextcloud-test", "urn:oid:151", null)
14. /var/www/html/lib/private/Files/ObjectStore/ObjectStoreStorage.php line 466
OC\Files\ObjectStore\Azure->writeObject("urn:oid:151", null)
15. /var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php line 631
OC\Files\ObjectStore\ObjectStoreStorage->writeStream("files/Documents ... x", null, null)
16. /var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php line 631
OC\Files\Storage\Wrapper\Wrapper->writeStream("files/Documents ... x", null, null)
17. /var/www/html/apps/dav/lib/Connector/Sabre/File.php line 202
OC\Files\Storage\Wrapper\Wrapper->writeStream("files/Documents ... x", null)
18. /var/www/html/apps/dav/lib/Connector/Sabre/Directory.php line 154
OCA\DAV\Connector\Sabre\File->put(null)
19. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 1104
OCA\DAV\Connector\Sabre\Directory->createFile("IT Security Rou ... x", null)
20. /var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 527
Sabre\DAV\Server->createFile("Documents/IT Se ... x", null, null)
21. /var/www/html/3rdparty/sabre/event/lib/WildcardEmitterTrait.php line 89
Sabre\DAV\CorePlugin->httpPut(Sabre\HTTP\Request {}, Sabre\HTTP\Response {})
22. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 474
Sabre\DAV\Server->emit("method:PUT", [Sabre\HTTP\Requ ... }])
23. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 251
Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Request {}, Sabre\HTTP\Response {})
24. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 319
Sabre\DAV\Server->start()
25. /var/www/html/apps/dav/appinfo/v1/webdav.php line 82
Sabre\DAV\Server->exec()
26. /var/www/html/remote.php line 167
require_once("/var/www/html/a ... p")
szaimen
commented
3 years ago Is this Issue still valid in NC21.0.3? If not, please close this issue. Thanks! :)
ghost
commented
3 years ago This issue has been automatically marked as stale because it has not had recent activity and seems to be missing some essential information. It will be closed if no further activity occurs. Thank you for your contributions.
camilasan
commented
1 year ago I think this is still valid on 25, just saw a report.
mahafree
commented
1 year ago I'm trying this on 27 and still get the same error.
TVC-ScS
commented
1 year ago The same here. I'm using 27.0.0 on PHP 8.2.7 It would be great if this problem could be solved. Is there any way to support you for resolving this?
kesselb
commented
1 year ago Index: lib/private/Files/ObjectStore/Azure.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/lib/private/Files/ObjectStore/Azure.php b/lib/private/Files/ObjectStore/Azure.php
--- a/lib/private/Files/ObjectStore/Azure.php (revision a7002edaa1cbdce9c9402bb54ab68b7f80d2f47a)
+++ b/lib/private/Files/ObjectStore/Azure.php (date 1688981954056)
@@ -101,11 +101,14 @@
}
public function writeObject($urn, $stream, string $mimetype = null) {
+ $buffer = fopen('php://temp/maxmemory:33554432', 'rwb+');
+ stream_copy_to_stream($stream, $buffer);
+
$options = new CreateBlockBlobOptions();
if ($mimetype) {
$options->setContentType($mimetype);
}
- $this->getBlobClient()->createBlockBlob($this->containerName, $urn, $stream, $options);
+ $this->getBlobClient()->createBlockBlob($this->containerName, $urn, $buffer, $options);
}
/**Made it work for me. The patch copies the input data into a different stream. Downside, we need a temporary file for it.
The Content-Length header is a hard requirement for Azure Blob. If you upload a file (without server side encryption) the input stream php://input is "forwarded" to the azure uploader. As documented here https://www.php.net/manual/en/wrappers.php.php php://input does not support stat, and therefore we cannot detect the size.
Afaict it works with server side encryption because the data is cached on the Nextcloud server, encrypted and uploaded to azure. The encrypted stream support stat and can return the size for the Content-Length header.
brosahay
commented
1 month ago Index: lib/private/Files/ObjectStore/Azure.php IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/lib/private/Files/ObjectStore/Azure.php b/lib/private/Files/ObjectStore/Azure.php --- a/lib/private/Files/ObjectStore/Azure.php (revision a7002edaa1cbdce9c9402bb54ab68b7f80d2f47a) +++ b/lib/private/Files/ObjectStore/Azure.php (date 1688981954056) @@ -101,11 +101,14 @@ } public function writeObject($urn, $stream, string $mimetype = null) { + $buffer = fopen('php://temp/maxmemory:33554432', 'rwb+'); + stream_copy_to_stream($stream, $buffer); + $options = new CreateBlockBlobOptions(); if ($mimetype) { $options->setContentType($mimetype); } - $this->getBlobClient()->createBlockBlob($this->containerName, $urn, $stream, $options); + $this->getBlobClient()->createBlockBlob($this->containerName, $urn, $buffer, $options); } /**Made it work for me. The patch copies the input data into a different stream. Downside, we need a temporary file for it.
The Content-Length header is a hard requirement for Azure Blob. If you upload a file (without server side encryption) the input stream
php://inputis "forwarded" to the azure uploader. As documented here https://www.php.net/manual/en/wrappers.php.phpphp://inputdoes not support stat, and therefore we cannot detect the size.Afaict it works with server side encryption because the data is cached on the Nextcloud server, encrypted and uploaded to azure. The encrypted stream support stat and can return the size for the Content-Length header.
though it works to remove the error, but if larger files like videos and stuff then downloading them back is not working. Basically, anything that gets split with this does not download back properly for me.
TLDR: Trying to use azure as primary storage. Without encryption, can't upload to azure at all. With encryption, uploads work but it duplicates all files with one encrypted and one unencrypted copy.
Steps to reproduce
Expected behaviour
Azure should be used, and files uploaded to blob storage
Actual behaviour
Without encryption module enabled:
Files fail to upload to azure, presumably because they are missing the content length header
With encryption module enabled:
Files upload just fine! Awesome!
Except that it's actually storing each file twice in the container - once encrypted, and once unencrypted. The unencrypted copy stays there even after completely removing the file from nextcloud. The encrypted copy does get properly deleted.
I suspect that enabling the encryption module changes how the file is handled, instead of being streamed to azure, it's encrypted and a new transfer to azure happens, and that one has the proper headers?
It's so close to working properly that it's driving me crazy!
Server configuration
Operating system: Ubuntu 18.04
Web server/Database/PHP version: Everything configured by snap
Nextcloud version: 15.0.4.0
Updated from an older Nextcloud/ownCloud or fresh install: Fresh
Where did you install Nextcloud from: snap
Signing status:
Signing status
``` No errors have been found. ```List of activated apps:
App list
``` Enabled: - accessibility: 1.1.0 - activity: 2.8.2 - cloud_federation_api: 0.1.0 - dav: 1.8.1 - encryption: 2.3.0 - federatedfilesharing: 1.5.0 - federation: 1.5.0 - files: 1.10.0 - files_external: 1.6.0 - files_pdfviewer: 1.4.0 - files_rightclick: 0.11.0 - files_sharing: 1.7.0 - files_texteditor: 2.7.0 - files_trashbin: 1.5.0 - files_versions: 1.8.0 - files_videoplayer: 1.4.0 - firstrunwizard: 2.4.0 - gallery: 18.2.0 - keeporsweep: 0.2.1 - logreader: 2.0.0 - lookup_server_connector: 1.3.0 - nextcloud_announcements: 1.4.0 - notes: 2.5.1 - notifications: 2.3.0 - oauth2: 1.3.0 - password_policy: 1.5.0 - provisioning_api: 1.5.0 - quota_warning: 1.4.0 - serverinfo: 1.5.0 - sharebymail: 1.5.0 - support: 1.0.0 - survey_client: 1.3.0 - systemtags: 1.5.0 - twofactor_backupcodes: 1.4.1 - twofactor_u2f: 2.1.1 - workflowengine: 1.5.0 Disabled: - admin_audit - comments - cospend - files_downloadactivity - files_external_gdrive - files_markdown - files_mindmap - metadata - news - polls - quicknotes - ransomware_protection - tasks - terms_of_service - theming - twofactor_totp - user_ldap ```Nextcloud configuration:
Config report
``` { "system": { "objectstore": { "class": "OC\\Files\\ObjectStore\\Azure", "arguments": { "account_name": "myazurestorageaccount", "account_key": "myazurestoragekey", "container": "nameofcontainer" } }, "apps_paths": [ { "path": "\/snap\/nextcloud\/current\/htdocs\/apps", "url": "\/apps", "writable": false }, { "path": "\/var\/snap\/nextcloud\/current\/nextcloud\/extra-apps", "url": "\/extra-apps", "writable": true } ], "supportedDatabases": [ "mysql" ], "memcache.locking": "\\OC\\Memcache\\Redis", "memcache.local": "\\OC\\Memcache\\Redis", "redis": { "host": "***REMOVED SENSITIVE VALUE***", "port": 0 }, "instanceid": "***REMOVED SENSITIVE VALUE***", "passwordsalt": "***REMOVED SENSITIVE VALUE***", "secret": "***REMOVED SENSITIVE VALUE***", "trusted_domains": [ "cloud.emsley.ca" ], "datadirectory": "***REMOVED SENSITIVE VALUE***", "dbtype": "mysql", "version": "15.0.4.0", "overwrite.cli.url": "http:\/\/my.server.com", "dbname": "***REMOVED SENSITIVE VALUE***", "dbhost": "***REMOVED SENSITIVE VALUE***", "dbport": "", "dbtableprefix": "oc_", "mysql.utf8mb4": true, "dbuser": "***REMOVED SENSITIVE VALUE***", "dbpassword": "***REMOVED SENSITIVE VALUE***", "installed": true, "mail_smtpmode": "smtp", "mail_smtpsecure": "tls", "mail_sendmailmode": "smtp", "mail_from_address": "***REMOVED SENSITIVE VALUE***", "mail_domain": "***REMOVED SENSITIVE VALUE***", "mail_smtpauthtype": "PLAIN", "mail_smtpauth": 1, "mail_smtphost": "***REMOVED SENSITIVE VALUE***", "mail_smtpport": "587", "mail_smtpname": "***REMOVED SENSITIVE VALUE***", "mail_smtppassword": "***REMOVED SENSITIVE VALUE***", "maintenance": false, "twofactor_enforced": "false", "twofactor_enforced_groups": [], "twofactor_enforced_excluded_groups": [] } } ```Are you using external storage, if yes which one: Azure
Are you using encryption: Trying to
Are you using an external user-backend, if yes which one: No
Client configuration
Browser: Tried Chrome and Firefox
Operating system: Windows 10
Logs
Web server error log
Web server error log
No relevant errorsNextcloud log (data/nextcloud.log)
Nextcloud log
See above for the relevant errorsBrowser log
Browser log
No relevant errors