search

Jumoo / uSync.Complete.Issues

Public Issue tracker and roadmap for uSync.Complete
https://jumoo.co.uk/usync/complete/
2 stars 1 forks source link

Bug Copying Media with AWS S3 #75

Closed Attackmonkey closed 3 years ago

Attackmonkey commented 3 years ago

Description We have two sites both in V8, both are set up to use the S3 provider for the files. On the one site, we have everything set to get saved into /media in the bucket prefix settings for the provider. On the other, the prefix is /PR1802/media (something to do with the client setup). This breaks the back office image linking, which seems to be an issue with either core or the S3 provider (still trying to work that out).

However, if we then try and copy media from one environment to the other, it only half succeeds. So if we export or import, it looks like it's worked, and it creates the new nodes in the media section, but the files are all missing. In the site logs, we get the error "Could not populate upload auto-fill properties for file '{File}'.". the stack trace is as follows:

System.IO.FileNotFoundException: The specified key does not exist. ---> Amazon.S3.AmazonS3Exception: The specified key does not exist. ---> Amazon.Runtime.Internal.HttpErrorResponseException: The remote server returned an error: (404) Not Found. ---> System.Net.WebException: The remote server returned an error: (404) Not Found.
   at System.Net.HttpWebRequest.GetResponse()
   at Amazon.Runtime.Internal.HttpRequest.GetResponse() in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\HttpHandler\_bcl\HttpWebRequestFactory.cs:line 166
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.HttpRequest.GetResponse() in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\HttpHandler\_bcl\HttpWebRequestFactory.cs:line 178
   at Amazon.Runtime.Internal.HttpHandler`1.InvokeSync(IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\HttpHandler\HttpHandler.cs:line 85
   at Amazon.Runtime.Internal.RedirectHandler.InvokeSync(IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\RedirectHandler.cs:line 41
   at Amazon.Runtime.Internal.Unmarshaller.InvokeSync(IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\Unmarshaller.cs:line 50
   at Amazon.S3.Internal.AmazonS3ResponseHandler.InvokeSync(IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Services\S3\Custom\Internal\AmazonS3ResponseHandler.cs:line 44
   at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\ErrorHandler\ErrorHandler.cs:line 72
   --- End of inner exception stack trace ---
   at Amazon.Runtime.Internal.HttpErrorResponseExceptionHandler.HandleException(IExecutionContext executionContext, HttpErrorResponseException exception) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\ErrorHandler\HttpErrorResponseExceptionHandler.cs:line 60
   at Amazon.Runtime.Internal.ErrorHandler.ProcessException(IExecutionContext executionContext, Exception exception) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\ErrorHandler\ErrorHandler.cs:line 204
   at Amazon.Runtime.Internal.ErrorHandler.InvokeSync(IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\ErrorHandler\ErrorHandler.cs:line 78
   at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\CallbackHandler.cs:line 46
   at Amazon.Runtime.Internal.EndpointDiscoveryHandler.InvokeSync(IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\EndpointDiscoveryHandler.cs:line 54
   at Amazon.Runtime.Internal.RetryHandler.InvokeSync(IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\RetryHandler\RetryHandler.cs:line 88
   at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\CallbackHandler.cs:line 46
   at Amazon.Runtime.Internal.CallbackHandler.InvokeSync(IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\CallbackHandler.cs:line 46
   at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeSync(IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Services\S3\Custom\Internal\AmazonS3ExceptionHandler.cs:line 48
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeSync(IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\ErrorCallbackHandler.cs:line 44
   at Amazon.Runtime.Internal.MetricsHandler.InvokeSync(IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\Handlers\MetricsHandler.cs:line 40
   at Amazon.Runtime.Internal.RuntimePipeline.InvokeSync(IExecutionContext executionContext) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\Pipeline\RuntimePipeline.cs:line 136
   at Amazon.Runtime.AmazonServiceClient.Invoke[TResponse](AmazonWebServiceRequest request, InvokeOptionsBase options) in D:\JenkinsWorkspaces\trebuchet-stage-release\AWSDotNetPublic\sdk\src\Core\Amazon.Runtime\AmazonServiceClient.cs:line 203
   at Umbraco.Storage.S3.BucketFileSystem.<>c__DisplayClass25_0.<OpenFile>b__0(IAmazonS3 client)
   at Umbraco.Storage.S3.BucketFileSystem.Execute[T](Func`2 request)
   at Umbraco.Storage.S3.BucketFileSystem.Execute[T](Func`2 request)
   at Umbraco.Storage.S3.BucketFileSystem.OpenFile(String path)
   at Umbraco.Web.Media.UploadAutoFillProperties.Populate(IContentBase content, IImagingAutoFillUploadField autoFillConfig, String filepath, String culture, String segment) in D:\a\1\s\src\Umbraco.Web\Media\UploadAutoFillProperties.cs:line 68

To Reproduce Steps to reproduce the behavior:

  1. install the V8 S3 provider: https://github.com/DannerrQ/Umbraco-S3-Provider on two sites, that are set up to communicate in uSync
  2. set up two S3 buckets
  3. configure both sites to use one of buckets, but set the "BucketFileSystem:MediaPrefix" value to be "/media" in one and "/test/media" in the other (you may need to create the folders in the bucket as well)
  4. upload an image to the site that's set up with the normal prefix
  5. try and copy that across to the other site using uSync, the UI will look like it succeeded, but if you expand the report out, there should be the failed file message
  6. try it the other way, and you should get the same result

Expected behavior I'd expect the media files to be successfully copied from one site to the other. If we have both sites set to use the standard /media prefix for the S3 provider, it works as expected.

Additional context Hopefully that all makes sense!

Upon further inspection, when pulling from the normal site to the one two folders deep, the media DOES get copied, but to the wrong place, it ends up in /test/media/media/normal-umbraco-file-stuff instead of being copied to /test/media/normal-umbraco-stuff. When copying the other way from the two deep prefix to the single one, it says it's copied, but nothing is copied across.

KevinJump commented 3 years ago

Hi,

Yes there is a similar thing for Azure blob storage too - to work with that we map the image path when importing and exporting media depending on the settings in the web.config.

The ImagePathMapper does this part. https://github.com/KevinJump/uSync8/blob/v8/8.7-main/uSync8.ContentEdition/Mapping/Mappers/ImagePathMapper.cs

There we look for a is a uSync.MediaFolder setting which details the path used locally. but if that isn't there we then look for the azure blob setting AzureBlobFileSystem.UseDefaultRoute:media and use that. (so for people using azure there is no extra config).

https://github.com/KevinJump/uSync8/blob/v8/8.7-main/uSync8.ContentEdition/Mapping/Mappers/ImagePathMapper.cs#L149

so you can set 'uSync.MediaFolder' and it should work.

but i think if we add BucketFileSystem:MediaPrefix to the list of look ups in this file then for future releases it will work without extra config?

Attackmonkey commented 3 years ago

Hi Kevin!

So, adding that setting fixes pulling down content from the /media server, to the /folder/media one, however, going the other way, we still get the same error. uSync says it's copied, and the item appears in the tree, with the correct media path, but the file isn't in the bucket and we still get the "Could not populate upload auto-fill properties for file '{File}'." error that we were getting going both ways previously. I've tried setting the folder to /media on the other server, and that doesn't seem to make any difference.

Attackmonkey commented 3 years ago

And some further updates. It doesn't seem to matter whether the folder is two deep or not. If your media prefix for AWS is not "media", you can pull from "/media" to "/something" as expected, but pushing the other way from "/something" to "/media" fails, with the same error as with two folders deep.

KevinJump commented 3 years ago

thanks for the info. i think i need to go install myself a S3 setup. and see if i can step through it. (i have my suspicions)

KevinJump commented 3 years ago

Hi,

I am having a little trouble setting up s3 storage with the alternative folder names :( -

I have this config: 

    <add key="BucketFileSystem:Region" value="eu-west-2" />
    <add key="BucketFileSystem:BucketName" value="jumoo-source" />
    <add key="BucketFileSystem:MediaPrefix" value="source" />
    <add key="BucketFileSystem:FormsPrefix" value="forms" />
    <add key="BucketFileSystem:BucketHostname" value="" />
    <add key="BucketFileSystem:DisableVirtualPathProvider" value="false" />

Also added the location setting to web.config for media, and the extra line in the /media/web.config folder.

However when i upload a file it does go to the s3 bucket, but umbraco is trying to access /source/folder/file.png. so it doesn't show up in the back end.

if i got to /media/folder/file.png it is there . so i think I've just gotten something small wrong 🤷

do i need to do something else when the prefix is different ?

Attackmonkey commented 3 years ago

Hiya, I solved that one last night. The S3 provider doesn't set the virtual file system to match the bucket prefix for some reason it always sets it to /media, I had to do a custom build of the S3 provider to make it honor the prefix. That solves that issue, you'll need to create the folder with the associated web.config file for the virtual file provider to work (the same as you normally do with the /media folder). That'll make the images work in the back office again!

The line of code I changed was: FileSystemVirtualPathProvider.ConfigureMedia(); to FileSystemVirtualPathProvider.ConfigureMedia(config.BucketPrefix); in the file BucketMediaFileSystemComponent.

After you've fixed that, you'll get the same errors with pushing from the renamed site! Let me know if you run into any other issues!

KevinJump commented 3 years ago

Hi,

I think i have fix for this - it was to do with how we calculate the generic path and then the path for the stream of the media item from disk (which isn't generic!).

So on my test i have gone from /source to /target folders and /source to /media folders

I do need to make sure i haven't changed anything to break how AzureBlob provider does this (i suspect i've also fixed the same unreported issue in azure).

It would be good if you could test this for me ?

what would be the best way to get you a test version of the main uSync.Expansions.Core.dll ? i can package it up if needed, but just replacing this dll on source/target sites should get it working. uSync.Expansions.Core.dll.zip

KevinJump commented 3 years ago

also worth noting, i think it only works when the / is in the uSync.Folder name (e.g /source and not source)

Attackmonkey commented 3 years ago

Cool, thanks! I'll grab that down and give that a go this afternoon, I'll let you know if it works.

Attackmonkey commented 3 years ago

OK, I've given that a quick test and it looks like it works!

:)

I'll do some more in depth testing later on.

Attackmonkey commented 3 years ago

OK, I've done some more in depth testing, and that all seems to be working perfectly now! I think we have a winner on the S3 front. Thanks for getting this sorted so quickly.

Attackmonkey commented 3 years ago

Do you know when you'll be able to do a release with the fix in? I can use the DLL you've sent for now though!

KevinJump commented 3 years ago

Hi, hopefully next week - i need to double check we haven't done any damage to azure - but this is the only fix lined up for v8.8 at the moment so shouldn't be to long to test.

KevinJump commented 3 years ago

The v8.8.1 release is complete and published now