httptest does not work with future multiprocess

[justynafurman]

I have a function that requests some data from an API using this piece of code:

df %>%
        mutate(new_column = future_map(id, function(x) {
          GET(create_request(x), authenticate(username, password, type = "basic"))
        },
        .progress = TRUE))

where create_request(x) creates a string with endpoint to call e.g. https://my_api.com/endpoint1/x. The code like that works just fine in with_mock_api function but, if I put plan(multiprocess) at the beginning I get an error

Could not resolve host: my_api.com

Is there any way to make it work with httptest in parallel with furrr package?

[nealrichardson]

Hmm, hard to tell what's going on. Could you create a minimal reproducible example (https://reprex.tidyverse.org/ or otherwise) so that I could run locally and see what you're seeing? Looks like you're pretty close to it with your code example, just need to simplify a bit and wrap it up.

[justynafurman]

@nealrichardson here it it a compressed file with minimum example and mocked api example.zip

[nealrichardson]

Thanks. I'll try to take a look in the next week or two. I haven't used future but my hunch is that the multiprocess setup is spawning new R processes, and those don't inherit the tracing/mocking from the parent process.

Out of curiosity, what is your use case for using future in this way? And how feasible would it be to turn off the multiprocessing for the API tests?

[justynafurman]

Actually, it's easy to turn off multiprocessing in my case, so it's not a big problem, but I wanted to address this issue anyway.