fetch apis not working from web worker thread in async mode

[alshamma]

Using the emscripten fetch methods from a thread hangs. If run code from main thread it works. Setting the EMSCRIPTEN_FETCH_SYNCHRONOUS flag allows the code to run from a thread. Unfortunately, our code expects http calls to be async, so we cannot use this solution.

Is this expected behavior? Is not what we expected based on the documentation.

[kripken]

This does not sound expected to me.

Trying to reproduce it, I did this:

diff --git a/tests/fetch/response_headers.cpp b/tests/fetch/response_headers.cpp
index ededdc161..9fa213217 100644
--- a/tests/fetch/response_headers.cpp
+++ b/tests/fetch/response_headers.cpp
@@ -22,7 +22,7 @@ int main()
     emscripten_fetch_attr_t attr;
     emscripten_fetch_attr_init( &attr );
     strcpy( attr.requestMethod, "GET" );
-    attr.attributes = EMSCRIPTEN_FETCH_REPLACE | EMSCRIPTEN_FETCH_LOAD_TO_MEMORY | EMSCRIPTEN_FETCH_SYNCHRONOUS;
+    attr.attributes = EMSCRIPTEN_FETCH_REPLACE | EMSCRIPTEN_FETCH_LOAD_TO_MEMORY;
     attr.requestHeaders = headers;

     attr.onsuccess = [] ( emscripten_fetch_t *fetch )
@@ -74,13 +74,4 @@ int main()
     };

     emscripten_fetch_t *fetch = emscripten_fetch( &attr, "gears.png" );
-    if ( result == 0 )
-    {
-        result = 2;
-        printf( "emscripten_fetch() failed to run synchronously!\n" );
-    }
-#ifdef REPORT_RESULT
-    // Fetch API appears to sometimes call the handlers more than once, see https://github.com/emscripten-core/emscripten/pull/8191
-    MAYBE_REPORT_RESULT(result);
-#endif
 }
diff --git a/tests/test_browser.py b/tests/test_browser.py
index b00b4932b..5df7e0a68 100644
--- a/tests/test_browser.py
+++ b/tests/test_browser.py
@@ -4306,7 +4306,7 @@ window.close = function() {
   @requires_threads
   def test_fetch_response_headers(self):
     shutil.copyfile(path_from_root('tests', 'gears.png'), 'gears.png')
-    self.btest('fetch/response_headers.cpp', expected='1', args=['-s', 'FETCH_DEBUG=1', '-s', 'FETCH=1', '-s', 'USE_PTHREADS=1', '-s', 'PROXY_TO_PTHREAD=1'], also_asmjs=True)
+    self.btest('fetch/response_headers.cpp', expected='1', args=['-s', 'FETCH_DEBUG=1', '-s', 'FETCH=1', '-s', 'USE_PTHREADS=1', '-s', 'PROXY_TO_PTHREAD=1'])

   # Test emscripten_fetch() usage to stream a XHR in to memory without storing the full file in memory
   def test_fetch_stream_file(self):

and then ran ./tests/runner.py browser.test_fetch_response_headers. That tests an async fetch in a worker, and it passes without error. So there may be something more specific in your use case.

Is an error shown in the console perhaps?

[alshamma]

Ah, I will have to investigate further. In a simple test app, fetch + threads does work. But, not in our larger app.

We get absolutely no output or error in the console. After xhr.send() is called from emscripten_fetch(), none of the events, onsuccess, onerror, etc. are called. Behaves like a hang.

[alshamma]

This code demonstrates the issue.

// Copyright 2016 The Emscripten Authors.  All rights reserved.
// Emscripten is available under two separate licenses, the MIT license and the
// University of Illinois/NCSA Open Source License.  Both these licenses can be
// found in the LICENSE file.

#include <stdio.h>
#include <string.h>
#include <iostream>
#include <thread>

#include <emscripten/fetch.h>

using TransferCompletionHandler = std::function<void()>;

static TransferCompletionHandler sCallback;

void downloadSucceeded(emscripten_fetch_t *fetch)
{
  printf("Finished downloading %llu bytes from URL %s.\n", fetch->numBytes, fetch->url);
  // The data is now available at fetch->data[0] through fetch->data[fetch->numBytes-1];
  emscripten_fetch_close(fetch); // Free data associated with the fetch.
  // auto callback = (TransferCompletionHandler) fetch->userData;
  auto callback = sCallback;
  callback();
}

void downloadFailed(emscripten_fetch_t *fetch)
{
  printf("Downloading %s failed, HTTP failure status code: %d.\n", fetch->url, fetch->status);
  emscripten_fetch_close(fetch); // Also free data on failure.
  // auto callback = (TransferCompletionHandler) fetch->userData;
  auto callback = sCallback;
  callback();
}

void invoke_fetch(TransferCompletionHandler callback) {
  emscripten_fetch_attr_t attr;
  emscripten_fetch_attr_init(&attr);
  sCallback = callback;
  strcpy(attr.requestMethod, "GET");
  attr.attributes = EMSCRIPTEN_FETCH_LOAD_TO_MEMORY;
  attr.onsuccess = downloadSucceeded;
  attr.onerror = downloadFailed;
  emscripten_fetch(&attr, "https://platform-cs-stage.adobe.io");  
}

void test_fetch()
{
    bool waiting{true};
    invoke_fetch([&](){
      std::cout << "invoke_fetch done" << std::endl;
      waiting = false;
    });
    while (waiting) {
        std::cout << "." << std::endl;
        std::this_thread::sleep_for(std::chrono::milliseconds(100));
    }
}

int main()
{  
  std::thread test(test_fetch);

  test.join();
  return 0;
}

[alshamma]

@kripken If I remove the wait loop, then the fetch callbacks work. The question then is how to i do a wait without blocking the js code.

[stale[bot]]

This issue has been automatically marked as stale because there has been no activity in the past year. It will be closed automatically if no further activity occurs in the next 30 days. Feel free to re-open at any time if this issue is still relevant.

[tonygrue]

I've run into the same issue/question and this is the only thread I can find on it.

I assumed callingemscripten_current_thread_process_queued_calls would address it, similar to pumping messages in windows, but that seems not to work for me either.

[parbo]

I also have this issue.

[sbc100]

In order for async work to complete the thread needs to return to its event loop.

So rather than looping in test_fetch you would just return. The call to emscripten_fetch will ensure that the thread stays alive until the fetch is complete.

[keminming]

Wondering, if the caller like to wait for fetch return result synchronously with async api(weird but possible). How to workaround the issue? Can we dispatch the call to a dedicate thread, and that thread never block?

On Thu, Mar 30, 2023 at 2:24 PM Sam Clegg @.***> wrote:

In order for async work to complete the thread needs to return to its event loop.

So rather than looping in test_fetch you would just return. The call to emscripten_fetch will ensure that the thread stays alive until the fetch is complete.

— Reply to this email directly, view it on GitHub https://github.com/emscripten-core/emscripten/issues/12772#issuecomment-1490984709, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXTPY2J6ERJ6J3LXH7RYRDW6X2Y7ANCNFSM4TU5ZTSA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- Best regards

Ke Wang

[sbc100]

What your describing would only be possible with something like -sASYNCIFY such that the wasm computation could be suspended while the event loops runs. See https://emscripten.org/docs/porting/asyncify.html

[gdevillele]

Experiencing the same issue here... Async fetch from main thread works, but from other threads, the callback functions are not called at all. I am trying to find the reason of it.

[sbc100]

@gdevillele are you letting the event loop run in the thread you are in? i.e. are you returning from the thread entry point? Unless you do, the fetch callbacks cannot fire.

[gdevillele]

@gdevillele are you letting the event loop run in the thread you are in? i.e. are you returning from the thread entry point? Unless you do, the fetch callbacks cannot fire.

@sbc100 oh, thanks a lot for explaining this! I used to send requests in a thread that was an infinite loop (thus never returning). This looks like it's the issue you are mentioning. 🙂