FTP get request with `useStreaming: true` causes segfault

[tom-tan]

How to reproduce

I reproduce it with ldc 1.30.0 on Ubuntu 22.04 (in Docker container).

sample.d:

#!/usr/bin/env dub
/+ dub.sdl:
dependency "requests" version="~>2.0.9"
+/
void main()
{
    import requests;
    import std : copy, File, format;

    auto src = "ftp://ftp.ncbi.nlm.nih.gov/pubmed/updatefiles/README.txt";
    auto rq = Request();
    rq.useStreaming = true;
    rq.verbosity = 3;
    auto rs = rq.get(src);

    auto dstFile = File("README.txt", "wb");
    rs.receiveAsRange()
      .copy(dstFile.lockingBinaryWriter);
}

And run the following command:

$ chmod +x sample.d
$ ./sample.d

Expected behavior

README.txt is downloaded without errors.

Actual behavior

It causes segfault with the following messages:

$ ./sample.d 
< 220-
< 220-
<  This warning banner provides privacy and security notices consistent with 
< 220-
<  This warning banner provides privacy and security notices consistent with 
<  applicable federal laws, directives, and other federal guidance for accessing 
< 220-
<  This warning banner provides privacy and security notices consistent with 
<  applicable federal laws, directives, and other federal guidance for accessing 
<  this Government system, which includes all devices/storage media attached to 
(snip)
< 220 FTP Server ready.
> USER anonymous
< 331 Anonymous login ok, send your complete email address as your password
> PASS requests@
< 230 Anonymous access granted, restrictions apply
> PWD
< 257 "/" is the current directory
> CWD pubmed/updatefiles
< 250 CWD command successful
> TYPE I
< 200 Type set to I
> SIZE README.txt
< 213 4535
> PASV
< 227 Entering Passive Mode (165,112,9,229,195,83).
> RETR README.txt
< 150 Opening BINARY mode data connection for README.txt (4535 bytes)
< 226 Transfer complete
Program exited with code -11

I confirmed that a similar code with useStreaming: false works without errors.

[ikod]

Hello, @tom-tan

Thanks for report, will check it soon

[ikod]

hello, @tom-tan!

Can confirm the problem. Looks like something wrong happens with stream copy. Looking into it

btw, I get different message `

PASV < 227 Entering Passive Mode (130,14,250,13,195,113). RETR README.txt < 150 Opening BINARY mode data connection for README.txt (4535 bytes) < 226 Transfer complete CWD / core.exception.OutOfMemoryError@core/lifetime.d(126): Memory allocation failed `

[ikod]

hello, @tom-tan

I can't figure out what's going on. This code works for dmd both in debug and in release mode, and do not work for ldc2 only in debug mode... It's too weird for me :(

Some details: looks like receiveAsRange structure somehow become corrupted, but after few days of debugging I can't find where and how it could happen.

[tom-tan]

Thank you for investigating the problem.

I checked with several compilers and got different results :-( I got the results with Ubuntu 22.04 on docker 23.0.1 on WSL2 on Windows 11.

• dmd 2.102.0

  • It works without errors!

    $ ./sample.d
    ...
    > CWD /
    < 250 CWD command successful

• ldc 1.30.0

  • It fails with the following message as I reported above.

    $ ./sample.d
    ...
    > RETR README.txt
    < 150 Opening BINARY mode data connection for README.txt (4535 bytes)
    < 226 Transfer complete
    Program exited with code -11

• ldc 1.31.0

  • It fails with the following message. It is a little bit different from the result with ldc 1.30.0.

    $ ./sample.d
    ...
    < 226 Transfer complete
    > CWD /
    < 250 CWD command successful
    core.exception.OutOfMemoryError@core/lifetime.d(126): Memory allocation failed

• ldc 1.32.0

  • It fails with the following message as you reproduced above. It is a little bit different from the result with ldc 1.30.0 and ldc 1.31.0.

    $ ./sample.d
    ...
    < 226 Transfer complete
    > CWD /
    core.exception.OutOfMemoryError@core/lifetime.d(126): Memory allocation failed

[ikod]

Hello @tom-tan

can you please check same setup but with -b release?

I think that there can be some problem related to ldc optimisations

[tom-tan]

Thank you for your advice.

can you please check same setup but with -b release?

I checked and found that it works without errors with ldc 1.30.0, 1.31.0 and 1.32.0 with -b release.

$ dub run -b release --single sample.d
...
< 226 Transfer complete
> CWD /
< 250 CWD command successful

[ikod]

@tom-tan I'll close it as I do not know where to move with this strange problem