Connection using path style to a personal S3 (not amazon) fails.

[SergioBertolinSG]

Steps to reproduce:

1.- Enable external storage. 2.- Run a container using S3 protocol. I've used https://hub.docker.com/r/webhippie/minio/ with a created bucket inside.

Steps to use this container: Having a docker daemon running do:

2.1.- Get minio image: docker pull webhippie/minio

2.2.- Create a container: For example: docker run -t -i -d -p 39000:9000 webhippie/minio:latest This will drop and id.

Using docker logs id you can see some info of the container.

2.3.-Create a bucket: Access the external ip:redirected_port (for example 192.168.99.100:39000) using a browser to create it. You'll need the AccessKey and SecretKey from docker logs.

3.- Set up an external storage using S3

My config:

Expected behaviour

Connection works and you can upload files to it.

Actual behaviour

Connection fails and there are no logs.

Server configuration

Operating system: Ubuntu 16.04

Web server: Apache

Database: MySQL

PHP version: 7.0

ownCloud version: (see ownCloud admin page) Current master {"installed":true,"maintenance":false,"needsDbUpgrade":false,"version":"10.0.0.0","versionstring":"10.0.0","edition":"","productname":"ownCloud"}

Updated from an older ownCloud or fresh install: Fresh

The content of config/config.php:

Are you using external storage, if yes which one: local/smb/sftp/... Trying to use S3

Are you using encryption: No

Logs

No logs even when seeing it failing.

Client configuration

Firefox.

[SergioBertolinSG]

This a blocker for: https://github.com/owncloud/qa-enterprise/issues/55

[SergioBertolinSG]

No longer a blocker. Objectstore works with that container.

[PVince81]

Did you check the apache error log maybe ?

[mdusher]

Is there a fix for this? I'm running into the same problem using a minio container. As I wasn't seeing any logs I added $this->logger->logException($e); into the catch (\Exception $e) block in apps/files_external/lib/Controller/StoragesController.php#270

From that, I get the following log which seems to be caused when trying to retrieve the Bucket ACLs (this part here) and that's as far as I've managed to look.

Log

  "reqId": "Sn1onWps7uYrjJYhqIx5",
  "level": 3,
  "time": "2017-12-20T23:10:22+00:00",
  "remoteAddr": "10.42.4.110",
  "user": "michael.usher",
  "app": "no app in context",
  "method": "GET",
  "url": "/plus/index.php/apps/files_external/userstorages/24?testOnly=false",
  "message": "Exception: {\"Exception\":\"Aws\\\\S3\\\\Exception\\\\NotImplementedException\",\"Message\":\"A header you provided implies functionality that is not implemented\",\"Code\":0,\"Trace\":\"#0 \\/cloudstor\\/www\\/owncloud\\/apps\\/files_external\\/3rdparty\\/aws-sdk-php\\/Aws\\/Common\\/Exception\\/NamespaceExceptionFactory.php(76): Aws\\\\Common\\\\Exception\\\\NamespaceExceptionFactory->createException('Aws\\\\\\\\S3\\\\\\\\Exceptio...', Object(Guzzle\\\\Http\\\\Message\\\\Request), Object(Guzzle\\\\Http\\\\Message\\\\Response), Array)\\n#1 \\/cloudstor\\/www\\/owncloud\\/apps\\/files_external\\/3rdparty\\/aws-sdk-php\\/Aws\\/Common\\/Exception\\/ExceptionListener.php(55): Aws\\\\Common\\\\Exception\\\\NamespaceExceptionFactory->fromResponse(Object(Guzzle\\\\Http\\\\Message\\\\Request), Object(Guzzle\\\\Http\\\\Message\\\\Response))\\n#2 [internal function]: Aws\\\\Common\\\\Exception\\\\ExceptionListener->onRequestError(Object(Guzzle\\\\Common\\\\Event), 'request.error', Object(Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher))\\n#3 \\/cloudstor\\/www\\/owncloud\\/lib\\/composer\\/symfony\\/event-dispatcher\\/EventDispatcher.php(174): call_user_func(Array, Object(Guzzle\\\\Common\\\\Event), 'request.error', Object(Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher))\\n#4 \\/cloudstor\\/www\\/owncloud\\/lib\\/composer\\/symfony\\/event-dispatcher\\/EventDispatcher.php(43): Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher->doDispatch(Array, 'request.error', Object(Guzzle\\\\Common\\\\Event))\\n#5 \\/cloudstor\\/www\\/simplesaml\\/vendor\\/guzzle\\/guzzle\\/src\\/Guzzle\\/Http\\/Message\\/Request.php(589): Symfony\\\\Component\\\\EventDispatcher\\\\EventDispatcher->dispatch('request.error', Object(Guzzle\\\\Common\\\\Event))\\n#6 \\/cloudstor\\/www\\/simplesaml\\/vendor\\/guzzle\\/guzzle\\/src\\/Guzzle\\/Http\\/Message\\/Request.php(378): Guzzle\\\\Http\\\\Message\\\\Request->processResponse(Array)\\n#7 \\/cloudstor\\/www\\/simplesaml\\/vendor\\/guzzle\\/guzzle\\/src\\/Guzzle\\/Http\\/Curl\\/CurlMulti.php(302): Guzzle\\\\Http\\\\Message\\\\Request->setState('complete', Array)\\n#8 \\/cloudstor\\/www\\/simplesaml\\/vendor\\/guzzle\\/guzzle\\/src\\/Guzzle\\/Http\\/Curl\\/CurlMulti.php(257): Guzzle\\\\Http\\\\Curl\\\\CurlMulti->processResponse(Object(Guzzle\\\\Http\\\\Message\\\\Request), Object(Guzzle\\\\Http\\\\Curl\\\\CurlHandle), Array)\\n#9 \\/cloudstor\\/www\\/simplesaml\\/vendor\\/guzzle\\/guzzle\\/src\\/Guzzle\\/Http\\/Curl\\/CurlMulti.php(240): Guzzle\\\\Http\\\\Curl\\\\CurlMulti->processMessages()\\n#10 \\/cloudstor\\/www\\/simplesaml\\/vendor\\/guzzle\\/guzzle\\/src\\/Guzzle\\/Http\\/Curl\\/CurlMulti.php(224): Guzzle\\\\Http\\\\Curl\\\\CurlMulti->executeHandles()\\n#11 \\/cloudstor\\/www\\/simplesaml\\/vendor\\/guzzle\\/guzzle\\/src\\/Guzzle\\/Http\\/Curl\\/CurlMulti.php(111): Guzzle\\\\Http\\\\Curl\\\\CurlMulti->perform()\\n#12 \\/cloudstor\\/www\\/simplesaml\\/vendor\\/guzzle\\/guzzle\\/src\\/Guzzle\\/Http\\/Curl\\/CurlMultiProxy.php(94): Guzzle\\\\Http\\\\Curl\\\\CurlMulti->send()\\n#13 \\/cloudstor\\/www\\/simplesaml\\/vendor\\/guzzle\\/guzzle\\/src\\/Guzzle\\/Http\\/Client.php(284): Guzzle\\\\Http\\\\Curl\\\\CurlMultiProxy->send()\\n#14 \\/cloudstor\\/www\\/owncloud\\/apps\\/files_external\\/3rdparty\\/aws-sdk-php\\/Aws\\/Common\\/Client\\/AbstractClient.php(262): Guzzle\\\\Http\\\\Client->send(Object(Guzzle\\\\Http\\\\Message\\\\Request))\\n#15 \\/cloudstor\\/www\\/simplesaml\\/vendor\\/guzzle\\/guzzle\\/src\\/Guzzle\\/Service\\/Client.php(136): Aws\\\\Common\\\\Client\\\\AbstractClient->send(Object(Guzzle\\\\Http\\\\Message\\\\Request))\\n#16 \\/cloudstor\\/www\\/simplesaml\\/vendor\\/guzzle\\/guzzle\\/src\\/Guzzle\\/Service\\/Command\\/AbstractCommand.php(153): Guzzle\\\\Service\\\\Client->execute(Object(Aws\\\\S3\\\\Command\\\\S3Command))\\n#17 \\/cloudstor\\/www\\/simplesaml\\/vendor\\/guzzle\\/guzzle\\/src\\/Guzzle\\/Service\\/Command\\/AbstractCommand.php(189): Guzzle\\\\Service\\\\Command\\\\AbstractCommand->execute()\\n#18 \\/cloudstor\\/www\\/simplesaml\\/vendor\\/guzzle\\/guzzle\\/src\\/Guzzle\\/Service\\/Client.php(76): Guzzle\\\\Service\\\\Command\\\\AbstractCommand->getResult()\\n#19 \\/cloudstor\\/www\\/owncloud\\/apps\\/files_external\\/3rdparty\\/aws-sdk-php\\/Aws\\/Common\\/Client\\/AbstractClient.php(107): Guzzle\\\\Service\\\\Client->__call('GetBucketAcl', Array)\\n#20 \\/cloudstor\\/www\\/owncloud\\/apps\\/files_external\\/lib\\/Lib\\/Storage\\/AmazonS3.php(545): Aws\\\\Common\\\\Client\\\\AbstractClient->__call('getBucketAcl', Array)\\n#21 \\/cloudstor\\/www\\/owncloud\\/lib\\/private\\/Files\\/External\\/LegacyUtil.php(202): OCA\\\\Files_External\\\\Lib\\\\Storage\\\\AmazonS3->test(false, 'false')\\n#22 \\/cloudstor\\/www\\/owncloud\\/apps\\/files_external\\/lib\\/Controller\\/StoragesController.php(255): OC\\\\Files\\\\External\\\\LegacyUtil::getBackendStatus('\\\\\\\\OCA\\\\\\\\Files_Exte...', Array, false, 'false')\\n#23 \\/cloudstor\\/www\\/owncloud\\/apps\\/files_external\\/lib\\/Controller\\/StoragesController.php(305): OCA\\\\Files_External\\\\Controller\\\\StoragesController->updateStorageStatus(Object(OC\\\\Files\\\\External\\\\StorageConfig), 'false')\\n#24 \\/cloudstor\\/www\\/owncloud\\/apps\\/files_external\\/lib\\/Controller\\/UserStoragesController.php(106): OCA\\\\Files_External\\\\Controller\\\\StoragesController->show('24', 'false')\\n#25 [internal function]: OCA\\\\Files_External\\\\Controller\\\\UserStoragesController->show('24', 'false')\\n#26 \\/cloudstor\\/www\\/owncloud\\/lib\\/private\\/AppFramework\\/Http\\/Dispatcher.php(159): call_user_func_array(Array, Array)\\n#27 \\/cloudstor\\/www\\/owncloud\\/lib\\/private\\/AppFramework\\/Http\\/Dispatcher.php(89): OC\\\\AppFramework\\\\Http\\\\Dispatcher->executeController(Object(OCA\\\\Files_External\\\\Controller\\\\UserStoragesController), 'show')\\n#28 \\/cloudstor\\/www\\/owncloud\\/lib\\/private\\/AppFramework\\/App.php(98): OC\\\\AppFramework\\\\Http\\\\Dispatcher->dispatch(Object(OCA\\\\Files_External\\\\Controller\\\\UserStoragesController), 'show')\\n#29 \\/cloudstor\\/www\\/owncloud\\/lib\\/private\\/AppFramework\\/Routing\\/RouteActionHandler.php(46): OC\\\\AppFramework\\\\App::main('OCA\\\\\\\\Files_Exter...', 'show', Object(OC\\\\AppFramework\\\\DependencyInjection\\\\DIContainer), Array)\\n#30 [internal function]: OC\\\\AppFramework\\\\Routing\\\\RouteActionHandler->__invoke(Array)\\n#31 \\/cloudstor\\/www\\/owncloud\\/lib\\/private\\/Route\\/Router.php(307): call_user_func(Object(OC\\\\AppFramework\\\\Routing\\\\RouteActionHandler), Array)\\n#32 \\/cloudstor\\/www\\/owncloud\\/lib\\/base.php(928): OC\\\\Route\\\\Router->match('\\/apps\\/files_ext...')\\n#33 \\/cloudstor\\/www\\/owncloud\\/index.php(56): OC::handleRequest()\\n#34 {main}\",\"File\":\"\\/cloudstor\\/www\\/owncloud\\/apps\\/files_external\\/3rdparty\\/aws-sdk-php\\/Aws\\/Common\\/Exception\\/NamespaceExceptionFactory.php\",\"Line\":91}"
}

[ownclouders]

Hey, this issue has been closed because the label status/STALE is set and there were no updates for 7 days. Feel free to reopen this issue if you deem it appropriate.

(This is an automated comment from GitMate.io.)

[mdusher]

I've been playing with this some more and narrowed it down to Minio not supporting bucket ACLs.

From what I can tell, the test() function in AmazonS3.php uses the getBucketAcls() function in the AWS SDK to test the S3 connection and because minio does not have this implemented it throws a NotImplementedException.

I ended up modifying the test() function to use listBuckets() from the AWS SDK instead and added an extra check in to make sure the bucket provided exists. My changes appear to work (I need to do some more testing and make sure it's still testing as much as it was previously before I submit a pull request) and can be found in this commit: https://github.com/mdusher/core/commit/0d9c175a0d0d5486390c01132be6956a8c6872f6

I was also considering doing a hybrid of my changes and the previous test - ie. try getBucketAcls(), if NotImplementedException is caught, try listBuckets().

[PVince81]

@mdusher if calling listBuckets() is compatible with any S3 version we could have that as catch-all solution. Thanks for your research!

[mdusher]

@PVince81 As far as getting a green dot on the Storages page goes, it seems to be. I've tested that much with AWS, NetApps' StorageGRID and Minio. I just need to make sure it doesn't affect anything else - I'm not sure if the ACLs are used anywhere else in files_external (quick look didn't seem like it, but I'll be on the safe side and look further into it tomorrow).

[DeepDiver1975]

I tried to make minio work last week as well -> yes some methods are not supported.

A more recent version of the s3 lib allowe to ask which operation is allowed. Updating the sdk seems to be reasonable anyhow ....

[DeepDiver1975]

minio support has been added with https://github.com/owncloud/files_external_s3/pull/2 ->close

[lock[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.