Question regarding .net with firebase storage emulator

[JBastiaan]

Hello,

im working on a project where i need to use Google Cloud Storage to persist files. Now i found firebase can be used as an emulator for Google Cloud Storage, as is also recommended in the docs.

However I'm having issues just uploading a basic file, code works fine on actual GCS, but the emulator call fails.

My setup looks like this:

Environment.SetEnvironmentVariable("STORAGE_EMULATOR_HOST", "localhost:9199");

var storageClient = await new StorageClientBuilder
{
    EmulatorDetection = EmulatorDetection.EmulatorOnly,
    UnauthenticatedAccess = true
}.BuildAsync();

var data = "{\"key\": \"value\"}"u8.ToArray();
using var stream = new MemoryStream(data);

//this fails against emulator
storageClient.UploadObject("default-bucket", "myfile.json", "application/octet-stream", stream);

//this call works against emulator
var obj = await storageClient.GetObjectAsync("default-bucket", "some_other_file_i_manually_uploaded",
    new GetObjectOptions());

The failing call fails with exceptions:

Google.GoogleApiException: The service storage has thrown an exception. HttpStatusCode is BadRequest. No error message was specified.

Newtonsoft.Json.JsonReaderException

Unexpected character encountered while parsing value: B. Path '', line 0, position 0.

The same code works when i run it with a different storageclient instance against actual Google Cloud Storage. How should i be debugging this issue? I've tried looking at logs in the firebase container, but i could not find anything

[jskeet]

Hmm... I can't reproduce this. Admittedly the version of the emulator I'm using is a fairly old one, but even so.

For more logging, at the start of your code add:

using Google;
using Google.Apis.Logging;
ApplicationContext.RegisterLogger(new ConsoleLogger(LogLevel.All));

And then after creating the client, add:

storageClient.Service.HttpClient.MessageHandler.LogEvents =
    LogEventType.RequestHeaders | LogEventType.ResponseBody;

That will let you see the response body containing the error.

[JBastiaan]

Hey, thanks for taking the time. I added your suggested logging, here's what i get in my console:

W2024-10-24 11:17:31.774838 Google.Apis.Requests.RequestBuilder Add parameter should not get null values. type=Query, name=key
D2024-10-24 11:17:31.816977 Google.Apis.Http.ConfigurableMessageHandler Request[00000001] Headers:
  [X-Upload-Content-Type] 'application/octet-stream'
  [X-Upload-Content-Length] '16'
  [User-Agent] 'gcloud-dotnet/4.10.0; google-api-dotnet-client/1.67.0.0; (gzip)'
  [x-goog-api-client] 'gl-dotnet/3.0.0 gccl/4.10.0 gax/4.8.0 gdcl/1.67.0.3365'
  [Content-Encoding] 'gzip'
  [Content-Type] 'application/json; charset=utf-8'
D2024-10-24 11:17:31.884231 Google.Apis.Http.ConfigurableMessageHandler Response[00000001] Body: 'Bad Request'
E2024-10-24 11:17:31.894376 Google.Apis.Upload.ResumableUpload MediaUpload - Exception occurred while initializing the upload The service storage has thrown an exception.
HttpStatusCode is BadRequest.
No JSON error details were specified.
Raw error details are: Bad Request
Google.GoogleApiException: The service storage has thrown an exception. HttpStatusCode is BadRequest. No error message was specified.
 ---> Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: B. Path '', line 0, position 0.
   at Newtonsoft.Json.JsonTextReader.ParseValue()
   at Newtonsoft.Json.JsonTextReader.Read()
   at Newtonsoft.Json.JsonReader.ReadAndMoveToContent()
   at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize(JsonReader reader, Type objectType)
   at Google.Apis.Json.NewtonsoftJsonSerializer.Deserialize(String input, Type type)
   at Google.Apis.Json.NewtonsoftJsonSerializer.Deserialize[T](String input)
   at Google.Apis.Responses.HttpResponseMessageExtensions.DeserializeErrorAsync(HttpResponseMessage response, String name, ISerializer serializer)
   --- End of inner exception stack trace ---
   at Google.Apis.Responses.HttpResponseMessageExtensions.DeserializeErrorAsync(HttpResponseMessage response, String name, ISerializer serializer)
   at Google.Apis.Upload.ResumableUpload.ExceptionForResponseAsync(HttpResponseMessage response)
   at Google.Apis.Upload.ResumableUpload`1.InitiateSessionAsync(CancellationToken cancellationToken)
   at Google.Apis.Upload.ResumableUpload.UploadAsync(CancellationToken cancellationToken)

Also, i wonder if maybe the issue is in how I'm running the firebase emulator, im running a docker-compose with an image i found like this. It seems to run fine, but maybe i should be using a different image? However i could not seem to find any official firebase emulator images? Or should one build their own image from a dockerfile? If so, is there some dockerfile that one can find in documentation or something to set this up?:

firebase-emulator:
  image: spine3/firebase-emulator
  container_name: firebase-emulator
  ports:
    - "4000:4000"
    - "9199:9199"
  environment:
    - GCP_PROJECT=myproj  

[jskeet]

Also, i wonder if maybe the issue is in how I'm running the firebase emulator, im running a docker-compose with an image i found like this.

That may well be relevant, yes. Can you try the same code running the Firebase emulator directly on your development machine? If it works in that situation, then it really sounds like it's a matter to discuss with Firebase folks.

(Questions about how you should run the Firebase emulator aren't really in scope for this repo, which is just for .NET client libraries.)

[JBastiaan]

Took me some time to get the emulator running on my machine, but yea once i did, the code seems to work fine. Ill see if i can get some help somewhere on the firebase side. Thanks

[jskeet]

Great. Will close this issue for now - if your investigations elsewhere end up suggesting a change we should make here, just add another cment with the details and we'll reopen it.