imazen / imageflow-dotnet-server

A super-fast image server to speed up your site - deploy as a microservice, serverless, or embeddable.
https://docs.imageflow.io
GNU Affero General Public License v3.0
246 stars 33 forks source link

HTTP 400 Bad Request (InvalidQueryParameterValue) with Azure blob retrieval #54

Closed atkulp closed 1 year ago

atkulp commented 2 years ago

I've got the latest Imageflow Server installed as part of an ASP.NET Core 3.1 app running in a Linux App Service but get errors on every request for an Azure blob resource.

Error:

2021-11-20T07:14:16.909114402Z       An unhandled exception has occurred while executing the request.
2021-11-20T07:14:16.910959703Z Azure.RequestFailedException: Value for one of the query parameters specified in the request URI is invalid.
2021-11-20T07:14:16.911004003Z RequestId:5705d9ae-e01e-0031-25de-dd191e000000
2021-11-20T07:14:16.911020003Z Time:2021-11-20T07:14:16.8552954Z
2021-11-20T07:14:16.911030803Z Status: 400 (Value for one of the query parameters specified in the request URI is invalid.)
2021-11-20T07:14:16.911041603Z ErrorCode: InvalidQueryParameterValue
2021-11-20T07:14:16.911052303Z
2021-11-20T07:14:16.911062903Z Additional Information:
2021-11-20T07:14:16.911073103Z QueryParameterName: comp
2021-11-20T07:14:16.911083303Z QueryParameterValue:
2021-11-20T07:14:16.911093503Z Reason:
2021-11-20T07:14:16.911103503Z
2021-11-20T07:14:16.911113603Z Headers:
2021-11-20T07:14:16.911123603Z Server: Microsoft-HTTPAPI/2.0
2021-11-20T07:14:16.911133703Z x-ms-request-id: 5705d9ae-e01e-0031-25de-dd191e000000
2021-11-20T07:14:16.911144103Z x-ms-client-request-id: f7cec28f-1203-4f89-b806-1582ba0e041e
2021-11-20T07:14:16.911154303Z x-ms-error-code: InvalidQueryParameterValue
2021-11-20T07:14:16.911183803Z Access-Control-Allow-Origin: *
2021-11-20T07:14:16.911194803Z Date: Sat, 20 Nov 2021 07:14:16 GMT
2021-11-20T07:14:16.911204103Z Content-Length: 351
2021-11-20T07:14:16.911213403Z Content-Type: application/xml

Stack trace:

2021-11-20T07:14:16.911231803Z    at Azure.Storage.Blobs.BlobRestClient.Blob.DownloadAsync_CreateResponse(ClientDiagnostics clientDiagnostics, Response response)
2021-11-20T07:14:16.911241703Z    at Azure.Storage.Blobs.BlobRestClient.Blob.DownloadAsync(ClientDiagnostics clientDiagnostics, HttpPipeline pipeline, Uri resourceUri, String version, String snapshot, String versionId, Nullable`1 timeout, String range, String leaseId, Nullable`1 rangeGetContentHash, Nullable`1 rangeGetContentCRC64, String encryptionKey, String encryptionKeySha256, Nullable`1 encryptionAlgorithm, Nullable`1 ifModifiedSince, Nullable`1 ifUnmodifiedSince, Nullable`1 ifMatch, Nullable`1 ifNoneMatch, String ifTags, String requestId, Boolean async, String operationName, CancellationToken cancellationToken)
2021-11-20T07:14:16.911253703Z    at Azure.Storage.Blobs.Specialized.BlobBaseClient.StartDownloadAsync(HttpRange range, BlobRequestConditions conditions, Boolean rangeGetContentHash, Int64 startOffset, Boolean async, CancellationToken cancellationToken)
2021-11-20T07:14:16.911264103Z    at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadInternal(HttpRange range, BlobRequestConditions conditions, Boolean rangeGetContentHash, Boolean async, CancellationToken cancellationToken)
2021-11-20T07:14:16.911274303Z    at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadAsync(HttpRange range, BlobRequestConditions conditions, Boolean rangeGetContentHash, CancellationToken cancellationToken)
2021-11-20T07:14:16.911285903Z    at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadAsync(CancellationToken cancellationToken)
2021-11-20T07:14:16.911296503Z    at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadAsync()
2021-11-20T07:14:16.911306203Z    at Imageflow.Server.Storage.AzureBlob.AzureBlobService.Fetch(String virtualPath)
2021-11-20T07:14:16.911315803Z    at Imageflow.Server.BlobFetchCache.GetBlob()
2021-11-20T07:14:16.911325203Z    at Imageflow.Server.ImageJobInfo.GetPrimaryBlobBytesAsync()
2021-11-20T07:14:16.911334703Z    at Imageflow.Server.ImageflowMiddleware.<>c__DisplayClass15_0.<<ProcessWithStreamCache>b__0>d.MoveNext()
2021-11-20T07:14:16.911345003Z --- End of stack trace from previous location where exception was thrown ---
2021-11-20T07:14:16.911354403Z    at Imazen.HybridCache.AsyncCache.<>c__DisplayClass35_0.<<GetOrCreateBytes>b__0>d.MoveNext()
2021-11-20T07:14:16.911364603Z --- End of stack trace from previous location where exception was thrown ---
2021-11-20T07:14:16.911373903Z    at Imazen.Common.Concurrency.AsyncLockProvider.TryExecuteAsync(String key, Int32 timeoutMs, CancellationToken cancellationToken, Func`1 success)
2021-11-20T07:14:16.911384003Z    at Imazen.HybridCache.AsyncCache.GetOrCreateBytes(Byte[] key, AsyncBytesResult dataProviderCallback, CancellationToken cancellationToken, Boolean retrieveContentType)
2021-11-20T07:14:16.911393903Z    at Imazen.HybridCache.HybridCache.GetOrCreateBytes(Byte[] key, AsyncBytesResult dataProviderCallback, CancellationToken cancellationToken, Boolean retrieveContentType)
2021-11-20T07:14:16.911412003Z    at Imageflow.Server.ImageflowMiddleware.ProcessWithStreamCache(HttpContext context, String cacheKey, ImageJobInfo info)
2021-11-20T07:14:16.911423003Z    at Imageflow.Server.ImageflowMiddleware.Invoke(HttpContext context)

I'm running:

ConfigureServices:

            services.AddHttpClient();

            services.AddCors();

            services.AddImageflowRemoteReaderService(new RemoteReaderServiceOptions
            {
                SigningKey = Configuration.GetValue<string>("AppConfiguration:ImageHostRemoteSigningKey")
            }
            .AddPrefix("/remote/"));

            // Make Azure files/containers available at /cloud
            services.AddImageflowAzureBlobService(
                new AzureBlobServiceOptions(
                        Configuration.GetValue<string>("AppConfiguration:StorageConnection"),
                        new BlobClientOptions())
                .MapPrefix("/cloud", "" ));

            services.AddImageflowHybridCache(new HybridCacheOptions(Path.Combine(env.ContentRootPath, "imagecache")));

I'm publishing as Release configuration, netcoreapp3.1, Framework-dependent, Portable. Blob storage is General-Purpose v1. I'm a licensed user, but this is a dev instance with a different domain. I've tried with both HTTP 422 and red dot enforcement but blob resources are always 404. The remote reader retrieves and scales images properly.

I can include the entire diagnostics page if additional detail is needed, but this seemed most important:

Diagnostics for Imageflow at XXXXX.azurewebsites.net generated 11/20/2021 07:32:08 UTC
libimageflow release 1.5.8-rc62 (1f6dc67ee* 2021-03-29) for x86_64-unknown-linux-gnu 
Please remember to provide this page when contacting support.
0 issues detected:

---------------------- License Validation OFF -----------------------
You have certified that you are complying with the AGPLv3 and have open-sourced your project at the following url:
https://i-need-a-license.com

You have chosen to abide by the AGPLv3, which can be found at https://www.gnu.org/licenses/agpl-3.0.en.html

Please ensure that the URL to your open source project is correctly registered via ImageflowMiddlewareOptions.SetMyOpenSourceProjectUrl().

No license keys found.

You are using EnforceLicenseWith.Http422Error. If there is a licensing error, HTTP status code 422 will be returned instead of serving the image. This can also be set to EnforceLicenseWith.RedDotWatermark.
For help with your license, email support@imazen.io and include the contents of this page.
---------------------- License Validation OFF -----------------------

Installed Plugins
Imageflow.Server.HybridCache.HybridCacheService
Imageflow.Server.Storage.RemoteReader.RemoteReaderService
Imageflow.Server.Storage.AzureBlob.AzureBlobService

Accepted querystring keys:

mode, anchor, flip, sflip, scale, cache, process, quality, zoom, dpr, crop, cropxunits, cropyunits, w, h, width, height, maxwidth, maxheight, format, thumbnail, autorotate, srotate, rotate, ignoreicc, stretch, webp.lossless, webp.quality, frame, page, subsampling, colors, f.sharpen, f.sharpen_when, down.colorspace, 404, bgcolor, paddingcolor, bordercolor, preset, floatspace, jpeg_idct_downscale_linear, watermark, s.invert, s.sepia, s.grayscale, s.alpha, s.brightness, s.contrast, s.saturation, trim.threshold, trim.percentpadding, a.blur, a.sharpen, a.removenoise, a.balancewhite, dither, jpeg.progressive, encoder, decoder, builder, s.roundcorners., paddingwidth, paddingheight, margin, borderwidth, decoder.min_precise_scaling_ratio

Accepted file extensions:

.png, .jpg, .jpeg, .jfif, .jif, .jfi, .jpe, .gif, .webp

Environment information:

Running on Unix 5.4.0.1056 and CLR 3.1.18 and .NET Core .NET Core 3.1.18
OS arch: 
lilith commented 2 years ago

It sounds like the request url to Azure is getting truncated. Or maybe the Azure API/libraries are mismatched?

I would use a network monitor to see exactly what URL is going to Azure, and make sure it is configured correctly.

On Sat, Nov 20, 2021, 12:36 AM Arian Kulp @.***> wrote:

I've got the latest Imageflow Server installed as part of an ASP.NET Core 3.1 app running in a Linux App Service but get errors on every request for an Azure blob resource.

Error:

2021-11-20T07:14:16.909114402Z An unhandled exception has occurred while executing the request. 2021-11-20T07:14:16.910959703Z Azure.RequestFailedException: Value for one of the query parameters specified in the request URI is invalid. 2021-11-20T07:14:16.911004003Z RequestId:5705d9ae-e01e-0031-25de-dd191e000000 2021-11-20T07:14:16.911020003Z Time:2021-11-20T07:14:16.8552954Z 2021-11-20T07:14:16.911030803Z Status: 400 (Value for one of the query parameters specified in the request URI is invalid.) 2021-11-20T07:14:16.911041603Z ErrorCode: InvalidQueryParameterValue 2021-11-20T07:14:16.911052303Z 2021-11-20T07:14:16.911062903Z Additional Information: 2021-11-20T07:14:16.911073103Z QueryParameterName: comp 2021-11-20T07:14:16.911083303Z QueryParameterValue: 2021-11-20T07:14:16.911093503Z Reason: 2021-11-20T07:14:16.911103503Z 2021-11-20T07:14:16.911113603Z Headers: 2021-11-20T07:14:16.911123603Z Server: Microsoft-HTTPAPI/2.0 2021-11-20T07:14:16.911133703Z x-ms-request-id: 5705d9ae-e01e-0031-25de-dd191e000000 2021-11-20T07:14:16.911144103Z x-ms-client-request-id: f7cec28f-1203-4f89-b806-1582ba0e041e 2021-11-20T07:14:16.911154303Z x-ms-error-code: InvalidQueryParameterValue 2021-11-20T07:14:16.911183803Z Access-Control-Allow-Origin: * 2021-11-20T07:14:16.911194803Z Date: Sat, 20 Nov 2021 07:14:16 GMT 2021-11-20T07:14:16.911204103Z Content-Length: 351 2021-11-20T07:14:16.911213403Z Content-Type: application/xml

Stack trace:

2021-11-20T07:14:16.911231803Z at Azure.Storage.Blobs.BlobRestClient.Blob.DownloadAsync_CreateResponse(ClientDiagnostics clientDiagnostics, Response response) 2021-11-20T07:14:16.911241703Z at Azure.Storage.Blobs.BlobRestClient.Blob.DownloadAsync(ClientDiagnostics clientDiagnostics, HttpPipeline pipeline, Uri resourceUri, String version, String snapshot, String versionId, Nullable1 timeout, String range, String leaseId, Nullable1 rangeGetContentHash, Nullable1 rangeGetContentCRC64, String encryptionKey, String encryptionKeySha256, Nullable1 encryptionAlgorithm, Nullable1 ifModifiedSince, Nullable1 ifUnmodifiedSince, Nullable1 ifMatch, Nullable1 ifNoneMatch, String ifTags, String requestId, Boolean async, String operationName, CancellationToken cancellationToken) 2021-11-20T07:14:16.911253703Z at Azure.Storage.Blobs.Specialized.BlobBaseClient.StartDownloadAsync(HttpRange range, BlobRequestConditions conditions, Boolean rangeGetContentHash, Int64 startOffset, Boolean async, CancellationToken cancellationToken) 2021-11-20T07:14:16.911264103Z at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadInternal(HttpRange range, BlobRequestConditions conditions, Boolean rangeGetContentHash, Boolean async, CancellationToken cancellationToken) 2021-11-20T07:14:16.911274303Z at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadAsync(HttpRange range, BlobRequestConditions conditions, Boolean rangeGetContentHash, CancellationToken cancellationToken) 2021-11-20T07:14:16.911285903Z at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadAsync(CancellationToken cancellationToken) 2021-11-20T07:14:16.911296503Z at Azure.Storage.Blobs.Specialized.BlobBaseClient.DownloadAsync() 2021-11-20T07:14:16.911306203Z at Imageflow.Server.Storage.AzureBlob.AzureBlobService.Fetch(String virtualPath) 2021-11-20T07:14:16.911315803Z at Imageflow.Server.BlobFetchCache.GetBlob() 2021-11-20T07:14:16.911325203Z at Imageflow.Server.ImageJobInfo.GetPrimaryBlobBytesAsync() 2021-11-20T07:14:16.911334703Z at Imageflow.Server.ImageflowMiddleware.<>cDisplayClass15_0.<b0>d.MoveNext() 2021-11-20T07:14:16.911345003Z --- End of stack trace from previous location where exception was thrown --- 2021-11-20T07:14:16.911354403Z at Imazen.HybridCache.AsyncCache.<>cDisplayClass35_0.<b0>d.MoveNext() 2021-11-20T07:14:16.911364603Z --- End of stack trace from previous location where exception was thrown --- 2021-11-20T07:14:16.911373903Z at Imazen.Common.Concurrency.AsyncLockProvider.TryExecuteAsync(String key, Int32 timeoutMs, CancellationToken cancellationToken, Func`1 success) 2021-11-20T07:14:16.911384003Z at Imazen.HybridCache.AsyncCache.GetOrCreateBytes(Byte[] key, AsyncBytesResult dataProviderCallback, CancellationToken cancellationToken, Boolean retrieveContentType) 2021-11-20T07:14:16.911393903Z at Imazen.HybridCache.HybridCache.GetOrCreateBytes(Byte[] key, AsyncBytesResult dataProviderCallback, CancellationToken cancellationToken, Boolean retrieveContentType) 2021-11-20T07:14:16.911412003Z at Imageflow.Server.ImageflowMiddleware.ProcessWithStreamCache(HttpContext context, String cacheKey, ImageJobInfo info) 2021-11-20T07:14:16.911423003Z at Imageflow.Server.ImageflowMiddleware.Invoke(HttpContext context)

I'm running:

  • Imageflow.AllPlatforms (0.7.24)
  • Imageflow.Server (0.5.10)
  • Imageflow.Server.HybridCache (0.5.10)
  • Imageflow.Server.Storage.AzureBlob (0.5.10)
  • Imageflow.Server.Storage.RemoteReader (0.5.10)
  • Imazen.Common (0.5.10)
  • Microsoft.AspNetCore.AzureAppServices.HostingStartup (3.1.21)
  • Microsoft.Extensions.Http.Polly (3.1.21)

ConfigureServices:

        services.AddHttpClient();

        services.AddCors();

        services.AddImageflowRemoteReaderService(new RemoteReaderServiceOptions
        {
            SigningKey = Configuration.GetValue<string>("AppConfiguration:ImageHostRemoteSigningKey")
        }
        .AddPrefix("/remote/"));

        // Make Azure files/containers available at /cloud
        services.AddImageflowAzureBlobService(
            new AzureBlobServiceOptions(
                    Configuration.GetValue<string>("AppConfiguration:StorageConnection"),
                    new BlobClientOptions())
            .MapPrefix("/cloud", "" ));

        services.AddImageflowHybridCache(new HybridCacheOptions(Path.Combine(env.ContentRootPath, "imagecache")));

I'm publishing as Release configuration, netcoreapp3.1, Framework-dependent, Portable. Blob storage is General-Purpose v1. I'm a licensed user, but this is a dev instance with a different domain. I've tried with both HTTP 422 and red dot enforcement but blob resources are always 404. The remote reader retrieves and scales images properly.

I can include the entire diagnostics page if additional detail is needed, but this seemed most important:

Diagnostics for Imageflow at XXXXX.azurewebsites.net generated 11/20/2021 07:32:08 UTC libimageflow release 1.5.8-rc62 (1f6dc67ee* 2021-03-29) for x86_64-unknown-linux-gnu Please remember to provide this page when contacting support. 0 issues detected:

---------------------- License Validation OFF ----------------------- You have certified that you are complying with the AGPLv3 and have open-sourced your project at the following url:https://i-need-a-license.com

You have chosen to abide by the AGPLv3, which can be found at https://www.gnu.org/licenses/agpl-3.0.en.html

Please ensure that the URL to your open source project is correctly registered via ImageflowMiddlewareOptions.SetMyOpenSourceProjectUrl().

No license keys found.

You are using EnforceLicenseWith.Http422Error. If there is a licensing error, HTTP status code 422 will be returned instead of serving the image. This can also be set to EnforceLicenseWith.RedDotWatermark. For help with your license, email @.*** and include the contents of this page. ---------------------- License Validation OFF -----------------------

Installed Plugins Imageflow.Server.HybridCache.HybridCacheService Imageflow.Server.Storage.RemoteReader.RemoteReaderService Imageflow.Server.Storage.AzureBlob.AzureBlobService

Accepted querystring keys:

mode, anchor, flip, sflip, scale, cache, process, quality, zoom, dpr, crop, cropxunits, cropyunits, w, h, width, height, maxwidth, maxheight, format, thumbnail, autorotate, srotate, rotate, ignoreicc, stretch, webp.lossless, webp.quality, frame, page, subsampling, colors, f.sharpen, f.sharpen_when, down.colorspace, 404, bgcolor, paddingcolor, bordercolor, preset, floatspace, jpeg_idct_downscale_linear, watermark, s.invert, s.sepia, s.grayscale, s.alpha, s.brightness, s.contrast, s.saturation, trim.threshold, trim.percentpadding, a.blur, a.sharpen, a.removenoise, a.balancewhite, dither, jpeg.progressive, encoder, decoder, builder, s.roundcorners., paddingwidth, paddingheight, margin, borderwidth, decoder.min_precise_scaling_ratio

Accepted file extensions:

.png, .jpg, .jpeg, .jfif, .jif, .jfi, .jpe, .gif, .webp

Environment information:

Running on Unix 5.4.0.1056 and CLR 3.1.18 and .NET Core .NET Core 3.1.18 OS arch:

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/imazen/imageflow-dotnet-server/issues/54, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA2LHYWBTNBAJYRZ3PGUZ3UM5FZPANCNFSM5INVLSWA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

atkulp commented 2 years ago

Basically, nothing is making it to Azure. Every request is for an empty URL:

GET https://XXXXX.blob.core.windows.net/ HTTP/1.1
Host: XXXXX.blob.core.windows.net
x-ms-version: 2020-02-10
x-ms-client-request-id: 97aee227-9181-4f74-a5c4-6921f0843f4c
x-ms-return-client-request-id: true
User-Agent: azsdk-net-Storage.Blobs/12.7.0 (.NET Core 3.1.20; Microsoft Windows 10.0.22504)
x-ms-date: Mon, 22 Nov 2021 00:11:02 GMT
Authorization: SharedKey XXXXX:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=
traceparent: 00-eb556f0e1944924595d2e5137edfaf11-439224dffc41104b-00

So I guess it's accurate that "comp" is empty since it's not specified at all.

I'm using Azure.Storage.Common (12.6.0) and Azure.Storage.Blobs (12.7.0). I even updated them to the newest (12.9.0 for Common and 12.10.0 for Blobs) but no difference.

I started regressing versions of the Storage.AzureBlob plugin and it works on 0.5.2 but nothing later. This works:

      <PackageReference Include="Imageflow.Server" Version="0.5.10" />
      <PackageReference Include="Imageflow.Server.HybridCache" Version="0.5.10" />
      <PackageReference Include="Imageflow.Server.Storage.AzureBlob" Version="0.5.2" />
      <PackageReference Include="Imageflow.Server.Storage.RemoteReader" Version="0.5.10" />

Any newer version of AzureBlob fails by requesting only a "slash" for a URL. I changed nothing except the versions of these assemblies to narrow it down.

I hope this helps.

lilith commented 2 years ago

If this is still occurring with the latest release, could you upload an example project where this happens?

lilith commented 1 year ago

No response, assuming issue was resolved.