Unix HTTP client "resolution failed: unknown scheme" if /etc/services is missing

[bbc2]

I made the following program:

let main =
  let%lwt _response =
    Cohttp_lwt_unix.Client.get (Uri.of_string "http://127.0.0.1:8000")
  in
  Lwt.return ()

let () =
  Lwt_main.run main

When I run it in an environment where the /etc/services file is missing, like the default Debian Docker image, it fails with the following message:

# ./main.exe 
Fatal error: exception (Failure "resolution failed: unknown scheme")

As far as I can tell, this is due to how Conduit resolves URLs with Lwt_unix.getservbyname.

Since Cohttp is an HTTP library, maybe it shouldn't depend on this database? Or maybe Lwt or Conduit should have an external dependency on netbase, the Debian package that provides the /etc/services file?

Versions used:

cohttp                  2.3.0
cohttp-lwt              2.3.0
cohttp-lwt-unix         2.3.0
conduit                 2.0.1
conduit-lwt             2.0.1
conduit-lwt-unix        2.0.1
lwt                     4.3.0
lwt_ppx                 1.2.3

[bbc2]

Actually, never mind my suggestion on depext: it's a run-time issue, not a compile-time one.

[shogochiai]

+1 I experienced the same and when I added the file, the error message has been changed.

[utdrmac]

Experienced same issue when working building another project. curl worked just fine. wget just fine. But using the URL through ocaml-cohttp failed "unknown scheme" until I added this file. Why is this file so critical? Why do other libraries/utilities not need this file?

[utdrmac]

In my docker ubuntu builds apt install netbase creates an /etc/services file. (FYI)

[mseri]

This should not be a problem from the next version of cohttp

[sternenseemann]

I'm hitting this again when running the tests for cohttp-lwt-unix in the nix build sandbox:

 test_parser alias cohttp-lwt-unix/test/runtest
..........
Ran: 10 tests in: 0.01 seconds.
OK
   test_body alias cohttp-lwt-unix/test/runtest
....
Ran: 4 tests in: 0.01 seconds.
OK
 test_sanity alias cohttp-lwt-unix/test/runtest (exit 1)
(cd _build/default/cohttp-lwt-unix/test && ./test_sanity.exe)
[DEBUG][application]: Cohttp debugging output is active
EEEEEEE
==============================================================================
Error: Cohttp Server Test:6:client_close

(Failure "resolution failed: unknown scheme")
------------------------------------------------------------------------------
==============================================================================
Error: Cohttp Server Test:5:expert response

(Failure "resolution failed: unknown scheme")
------------------------------------------------------------------------------
==============================================================================
Error: Cohttp Server Test:4:no leaks on requests

(Failure "resolution failed: unknown scheme")
------------------------------------------------------------------------------
==============================================================================
Error: Cohttp Server Test:3:massive chunked

(Failure "resolution failed: unknown scheme")
------------------------------------------------------------------------------
==============================================================================
Error: Cohttp Server Test:2:pipelined with interleaving requests

(Failure "resolution failed: unknown scheme")
------------------------------------------------------------------------------
==============================================================================
Error: Cohttp Server Test:1:pipelined chunk test

(Failure "resolution failed: unknown scheme")
------------------------------------------------------------------------------
==============================================================================
Error: Cohttp Server Test:0:sanity test

(Failure "resolution failed: unknown scheme")
------------------------------------------------------------------------------
Ran: 7 tests in: 0.00 seconds.
FAILED: Cases: 7 Tried: 7 Errors: 7 Failures: 0 Skip:  0 Todo: 0 Timeouts: 0.
test_sanity_noisy alias cohttp-lwt-unix/test/runtest (exit 1)
(cd _build/default/cohttp-lwt-unix/test && ./test_sanity_noisy.exe)
[DEBUG][application]: Cohttp debugging output is active
EEE
==============================================================================
Error: Cohttp Server Test:2:unreadable file returns 500

(Failure "resolution failed: unknown scheme")
------------------------------------------------------------------------------
==============================================================================
Error: Cohttp Server Test:1:no body when response is not modified

(Failure "resolution failed: unknown scheme")
------------------------------------------------------------------------------
==============================================================================
Error: Cohttp Server Test:0:empty chunk test

(Failure "resolution failed: unknown scheme")
------------------------------------------------------------------------------
Ran: 3 tests in: 0.00 seconds.
FAILED: Cases: 3 Tried: 3 Errors: 3 Failures: 0 Skip:  0 Todo: 0 Timeouts: 0.

Any ideas? I'm having trouble debugging this as I don't know where this error message originates.

[mseri]

I think this was resolved by conduit 3, since that has been pulled it is likely still an issue.

[mseri]

Which versions of conduit and cohttp are you testing?

[sternenseemann]

4.0.0 for both packages.

[mseri]

The culprit should be this call in conduit-lwt-unix mentioned in the report: https://github.com/mirage/ocaml-conduit/blob/a2f91d0fb17c9b0a8436dd39c4267b9144f0bdea/src/conduit-lwt-unix/resolver_lwt_unix.ml#L46

A temporary fix is to make sure you have the /etc/services file, I have asked to reopen the relative conduit issue as well

[zbaylin]

Sorry to revive a dead thread, but in case anyone else has this issue I think it's worth sharing. This error will also be shown if you reach the kernel's open file limit, which would mean cohttp can't open /etc/services. Not sure if this is still the case since I'm using 4.x, though

[acieroid]

I hit the same issue today. I deployed a static binary that uses cohttp, in a "scratch" docker. I had to manually add /etc/services. This is using cohttp 5.3.1.

[nmote]

I can also reproduce this. I am on Fedora 40 with cohttp 5.3.1 and cohttp-lwt-unix 5.3.0 (there is no 5.3.1 release). I copied the client tutorial code. With no changes to my system, it ran fine. I removed /etc/services and got the error:

Fatal error: exception Failure("resolution failed: unknown scheme")

curl https://www.reddit.com/ completes normally even with /etc/services removed.

[nmote]

As @mseri noted earlier in the thread, Conduit calls Lwt_unix.getservbyname. That function is just a wrapper around Unix.getservbyname. This, in turn, is just a wrapper around the C library function getservbyname, which requires /etc/services to be present.

While there is clearly precedent for this requirement, it seems that many other tools work just fine when /etc/services is not available, as is evidently the case in many barebones Docker images. It would be fantastic if this library could drop this requirement.