Open joerage opened 1 year ago
Thanks for reaching out @joerage. Just to be sure I understand this issue before jumping into it: you seem to emphasize the value of the message time-to-live, but nothing about the trace makes it immediately obvious to me why time-to-live may be at fault here. Can you please clarify this? Thanks!
Hi @davidmrdavid, I simply making this assumption because the call made to storage dataplane has a single query parameter: https://mystorage.queue.core.windows.net:443/devcenterjobs-control-00/messages?messagettl=%E2%88%921
@amdeel - do you have any advice on this? Perhaps we can add some more telemetry here to confirm that messagettl
is in fact the parameter that is invalid, as the exception doesn't make this fully clear. From there, the next steps in investigation would be clearer.
When you decode that URL, it does decode to -1
: %E2%88%921
== -1
. Is this code actually consistent (regardless of correctness)?
If this is happening on only one region, is it an issue with Azure storage in that region? @joerage, can you try crafting a request with this that query param on it and sending it to different regions of Azure Storage? Asking because I imagine you have a few instances of Azure Storage readily available to test it out. If not, we can try that soon.
Recommended actions:
messagettl
query param off entirely when it is -1
?We found some more information since my last post.
To your questions
Edit: We figured out the issue.
The problem is in the StorageAccount SDK, which uses long?.ToString()
without specifying an invariant culture. Our service is using a ASP.NET Core middleware to set current culture from the incoming request ([app.UseRequestLocalization()](https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.builder.applicationbuilderextensions.userequestlocalization?view=aspnetcore-6.0)
). We don't support all incoming cultures, just a subset. Still, in our case, one specific culture is causing the issue. But all the following culture will cause -1 to become encoded:
Culture: Estonian Culture: Estonian (Estonia) Culture: Basque Culture: Basque (Spain) Culture: Finnish Culture: Finnish (Finland) Culture: Faroese Culture: Faroese (Denmark) Culture: Faroese (Faroe Islands) Culture: Swiss German Culture: Swiss German (Switzerland) Culture: Swiss German (France) Culture: Swiss German (Liechtenstein) Culture: Colognian Culture: Colognian (Germany) Culture: Lithuanian Culture: Lithuanian (Lithuania) Culture: Norwegian Bokmål Culture: Norwegian Bokmål (Norway) Culture: Norwegian Bokmål (Svalbard & Jan Mayen) Culture: Norwegian Nynorsk Culture: Norwegian Nynorsk (Norway) Culture: Romansh Culture: Romansh (Switzerland) Culture: Northern Sami Culture: Northern Sami (Finland) Culture: Northern Sami (Norway) Culture: Northern Sami (Sweden) Culture: Slovenian Culture: Slovenian (Slovenia) Culture: Swedish Culture: Swedish (Åland Islands) Culture: Swedish (Finland) Culture: Swedish (Sweden)
Our solution is to set the culture to InvariantCulture right before starting the orchestration (and setting it back right after).
CultureInfo.CurrentCulture = CultureInfo.InvariantCulture;
To repro, simply set the culture before starting the orchestration e.g:
CultureInfo.CurrentCulture = new CultureInfo("sv");
CultureInfo.CurrentUICulture = new CultureInfo("sv");
var instance = await client.CreateOrchestrationInstanceAsync(name, version, input);
Here is a small program showing the issue (thanks @chrissmiller!)
using System;
using System.Collections.Generic;
using System.Globalization;
public class Program
{
public static void Main()
{
TimeSpan? timeToLive = TimeSpan.FromSeconds(-1);
long timeToLiveInSeconds = (long)timeToLive.Value.TotalSeconds;
// Display value using some standard format specifiers.
var fmts = new List<string>(){
"D"
};
var cultures = CultureInfo.GetCultures(CultureTypes.AllCultures);
foreach (CultureInfo culture in cultures){
foreach (string fmt in fmts){
test(timeToLiveInSeconds, fmt, culture);
}
}
}
public static void test(long val, string format, CultureInfo culture){
string a = val.ToString(format, culture);
if (Uri.EscapeUriString(a) == "%E2%88%921"){
//Console.WriteLine(format);
Console.WriteLine($"Culture: {culture.DisplayName}");
Console.WriteLine($"String value: {a}");
Console.WriteLine($"URI Escaped Value: {Uri.EscapeUriString(a)}");
}
}
}
Not certain where the fix is, but the issue doesn't appear to repro on the newer Storage SDKs (e.g., Azure.Storage.Queues)
Wow, amazing find @chrissmiller! @amdeel, @jviau: do we think we could implement a similar or equivalent trick to ignore the user-specified CultureInfo when encoding our requests to Azure Storage?
do we think we could implement a similar or equivalent trick to ignore the user-specified CultureInfo when encoding our requests to Azure Storage?
We're in the process of migrating to modern versions of the Azure Storage SDK. Assuming the underlying issue is fixed there, it might make sense to just rely on the SDK upgrade rather than pushing a short-lived fix to work around issues in the deprecated SDKs.
I believe it is addressed in the newer SDKs.
As for working around this in existing - two things to check:
@cgillum, I think the SDK upgrade will warrant a major version rev. We may want to consider fixing this in the existing version, just in case the migration from DurableTask.AzureStorage v1 to v2 is not frictionless and some customers are blocked. They may still benefit from this fix in v1. Not saying that is the case, but something to consider.
We are running in an issue with DTFx. Sometimes (and only in one region where our service is running - WestEurope), when enqueuing a queue message to start an orchestration, the queueing fails because of a bad parameter in the query string.
Sample query: https://mystorage.queue.core.windows.net:443/devcenterjobs-control-00/messages?messagettl=%E2%88%921
Error and stacktrace:
Looking at the code here, the value of the messagettl param should always be -1 (as we are not using NET462). And looking at storage client code, it should append it.
Any help to resolve this issue would be appreciated.