[SR-10252] swift build --sanitize=address failed

[swift-ci]

Previous ID	SR-10252
Radar	None
Original Reporter	Tof (JIRA User)
Type	Bug

Attachment: Download

Environment

Swift: 4.2.3 Linux: Ubuntu 16.04 Docker Desktop Community: 2.0.0.3 Vapor: 3.3.0

Additional Detail from JIRA

| | | |------------------|-----------------| |Votes | 0 | |Component/s | Compiler | |Labels | Bug | |Assignee | None | |Priority | Medium | md5: e55cd21289406cd52a9b80fe1c842ce6

Issue Description:

As describe on swift forum Memory leaking in Vapor app try to build a project with

swift build --sanitize=address

Steps to reproduce:

1. unzip test project (see attach file)

2. generate linux image with command

docker-compose build

3. deploy image on docker swarm

docker stack deploy -c docker-compose.yml testproject

4. attach to the image

docker exec -it $(docker ps -q -f name=testproject_ping) /bin/bash

5. build the project with

swift build --sanitize=address

Results:

Swift build generate this output:

root@534f8bac79de:/app# swift build --sanitize=address
Compile CNIOZlib empty.c
Compile CNIOSHA1 c_nio_sha1.c
Compile CNIOOpenSSL shims.c
Compile CNIOOpenSSL helpers.c
Compile CNIOLinux ifaddrs-android.c
Compile CNIOLinux shim.c
Compile Swift Module 'NIOPriorityQueue' (2 sources)
Compile Swift Module 'Debugging' (3 sources)
Compile Swift Module 'COperatingSystem' (1 sources)
Compile CNIOHTTPParser c_nio_http_parser.c
Compile CNIODarwin shim.c
Compile CNIOAtomics src/c-atomics.c
Compile CCryptoOpenSSL shim.c
Compile CBcrypt blf.c
Compile CBcrypt bcrypt.c
Compile CBase32 base32.c
Compile Swift Module 'NIOConcurrencyHelpers' (2 sources)
Compile Swift Module 'NIO' (55 sources)
Compile Swift Module 'NIOTLS' (3 sources)
Compile Swift Module 'Bits' (12 sources)
Compile Swift Module 'Async' (15 sources)
Compile Swift Module 'NIOFoundationCompat' (1 sources)
Compile Swift Module 'NIOHTTP1' (9 sources)
Compile Swift Module 'NIOOpenSSL' (17 sources)
Compile Swift Module 'Random' (4 sources)
Compile Swift Module 'Core' (25 sources)
Compile Swift Module 'NIOWebSocket' (9 sources)
Compile Swift Module 'Logging' (4 sources)
Compile Swift Module 'Multipart' (8 sources)
Compile Swift Module 'URLEncodedForm' (8 sources)
Compile Swift Module 'Service' (20 sources)
Compile Swift Module 'HTTP' (26 sources)
Compile Swift Module 'Validation' (18 sources)
Compile Swift Module 'Crypto' (19 sources)
Compile Swift Module 'Routing' (12 sources)
Compile Swift Module 'DatabaseKit' (30 sources)
Compile Swift Module 'TemplateKit' (41 sources)
Compile Swift Module 'Console' (28 sources)
Compile Swift Module 'WebSocket' (6 sources)
Compile Swift Module 'Command' (16 sources)
Compile Swift Module 'Vapor' (75 sources)
Compile Swift Module 'App' (4 sources)
Compile Swift Module 'Run' (1 sources)
Linking ./.build/x86_64-unknown-linux/debug/Run
/app/.build/checkouts/crypto.git-1444389797943994899/Sources/CBase32/base32.c:95: error: undefined reference to '__asan_version_mismatch_check_v6'
/app/.build/checkouts/crypto.git-1444389797943994899/Sources/CBcrypt/bcrypt.c:260: error: undefined reference to '__asan_version_mismatch_check_v6'
/app/.build/checkouts/crypto.git-1444389797943994899/Sources/CBcrypt/blf.c:657: error: undefined reference to '__asan_version_mismatch_check_v6'
/app/.build/checkouts/crypto.git-1444389797943994899/Sources/CCryptoOpenSSL/shim.c:29: error: undefined reference to '__asan_version_mismatch_check_v6'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
<unknown>:0: error: link command failed with exit code 1 (use -v to see invocation)
error: terminated(1): /usr/bin/swift-build-tool -f /app/.build/debug.yaml main output:

Expected results:

Build the project without errors

[weissi]

I believe the reason that you're seeing this issue is that the Swift compiler and the clang on your system have a different address sanitizer version an they're incompatible. That was a longstanding bug in Swift that it relied on the system's clang. So stuff (like ASan) that is relying on a particular clang version would only work if your system happened to have the right one. Swift 5 now finally fixed that by shipping a matching clang compiler :slight_smile:.

I can also confirm that the attached example project works just fine with ASan for me in Swift 5.

$ jw-docker-swift-5.0 swift build --sanitize=address
Fetching https://github.com/vapor/vapor.git
Fetching https://github.com/vapor/console.git
Fetching https://github.com/vapor/core.git
Fetching https://github.com/vapor/crypto.git
Fetching https://github.com/vapor/database-kit.git
Fetching https://github.com/vapor/http.git
Fetching https://github.com/vapor/multipart.git
Fetching https://github.com/vapor/routing.git
Fetching https://github.com/vapor/service.git
Fetching https://github.com/vapor/template-kit.git
Fetching https://github.com/vapor/url-encoded-form.git
Fetching https://github.com/vapor/validation.git
Fetching https://github.com/vapor/websocket.git
Fetching https://github.com/apple/swift-nio.git
Fetching https://github.com/apple/swift-nio-zlib-support.git
Fetching https://github.com/apple/swift-nio-ssl-support.git
Fetching https://github.com/apple/swift-nio-ssl.git
Completed resolution in 56.58s
Cloning https://github.com/vapor/websocket.git
Resolving https://github.com/vapor/websocket.git at 1.1.2
Cloning https://github.com/vapor/vapor.git
Resolving https://github.com/vapor/vapor.git at 3.3.0
Cloning https://github.com/vapor/http.git
Resolving https://github.com/vapor/http.git at 3.1.11
Cloning https://github.com/vapor/routing.git
Resolving https://github.com/vapor/routing.git at 3.0.2
Cloning https://github.com/vapor/console.git
Resolving https://github.com/vapor/console.git at 3.1.1
Cloning https://github.com/apple/swift-nio-ssl.git
Resolving https://github.com/apple/swift-nio-ssl.git at 1.4.0
Cloning https://github.com/vapor/url-encoded-form.git
Resolving https://github.com/vapor/url-encoded-form.git at 1.0.6
Cloning https://github.com/vapor/core.git
Resolving https://github.com/vapor/core.git at 3.7.3
Cloning https://github.com/vapor/validation.git
Resolving https://github.com/vapor/validation.git at 2.1.1
Cloning https://github.com/vapor/database-kit.git
Resolving https://github.com/vapor/database-kit.git at 1.3.3
Cloning https://github.com/apple/swift-nio.git
Resolving https://github.com/apple/swift-nio.git at 1.13.2
Cloning https://github.com/vapor/crypto.git
Resolving https://github.com/vapor/crypto.git at 3.3.2
Cloning https://github.com/apple/swift-nio-zlib-support.git
Resolving https://github.com/apple/swift-nio-zlib-support.git at 1.0.0
Cloning https://github.com/vapor/multipart.git
Resolving https://github.com/vapor/multipart.git at 3.0.3
Cloning https://github.com/apple/swift-nio-ssl-support.git
Resolving https://github.com/apple/swift-nio-ssl-support.git at 1.0.0
Cloning https://github.com/vapor/service.git
Resolving https://github.com/vapor/service.git at 1.0.2
Cloning https://github.com/vapor/template-kit.git
Resolving https://github.com/vapor/template-kit.git at 1.1.2
/Users/johannes/Downloads/test project/.build/checkouts/core/Sources/Core/Process+Execute.swift:163:17: warning: 'launchPath' is deprecated: renamed to 'executableURL'
        process.launchPath = path
                ^
/Users/johannes/Downloads/test project/.build/checkouts/core/Sources/Core/Process+Execute.swift:163:17: note: use 'executableURL' instead
        process.launchPath = path
                ^~~~~~~~~~
                executableURL
/Users/johannes/Downloads/test project/.build/checkouts/core/Sources/Core/Process+Execute.swift:167:17: warning: 'launch()' is deprecated: renamed to 'run'
        process.launch()
                ^
/Users/johannes/Downloads/test project/.build/checkouts/core/Sources/Core/Process+Execute.swift:167:17: note: use 'run' instead
        process.launch()
                ^~~~~~
                run
[44/44] Linking ./.build/x86_64-unknown-linux/debug/Run

[belkadan]

This means the version of Clang on your system is mismatched with the version of Swift, and therefore they expect a different ASan runtime. I think the Swift 5 toolchains have a Clang included with them to fix this issue!

cc @kubamracek

[swift-ci]

Comment by Christophe Braud (JIRA)

@weissi I confirm no problem with Swift 5
@belkadan if you look at my Dockerfile I use the official version of Swift 4.2.3 so logically clang and the swift compiler should have been good as it is now the case with Swift 5

As soon as I can I will recompile my project in Vapor 3 with Swift 5

[belkadan]

Sorry, I meant that before Swift 5, the Swift Docker image would have used a system version of Clang, which is almost certainly out of sync with the Swift compiler. I guess it would be reasonable for the package manager to not try to sanitize the C libraries in the process under those conditions, but I'm not sure that's worth changing in Swift 4.2.x now that Swift 5 is out. Or am I missing something?

[belkadan]

cc also @aciidb0mb3r

[swift-ci]

Comment by Christophe Braud (JIRA)

> Sorry, I meant that before Swift 5
@belkadan It's ok 🙂
We know now if we want to use sanitize we have to use Swift 5, that it!
Don't spent too much to time on this ticket
There is more important as the ByteBuffer improvement to avoid memory fragmentation in Swift NIO
By the way Johannes did an excellent investigation on this 👍 🙂

[weissi]

Tof (JIRA User) Just to comment here: The memory fragmentation isn't really a SwiftNIO issue. SwiftNIO offers all the tools to be more tighter in the case. However, there are a number of very straightforward cases where SwiftNIO could help the user through heuristics. For example if someone extracts a 10 byte Data from a 1MB ByteBuffer we shouldn't try to do zero-copy, we should just copy as it'll be better in all cases.

In the case of the Vapor database driver however, this wouldn't have helped. If all buffers are fairly large, the SwiftNIO can't automatically decide if a copy or zero-copy is better. That's up to the Vapor mysql driver really 🙂.

[swift-ci]

Comment by Christophe Braud (JIRA)

> The memory fragmentation isn't really a SwiftNIO issue. SwiftNIO offers all the tools to be more tighter in the case. However, there are a number of very straightforward cases where SwiftNIO could help the user through heuristics.

@weissi I understand your point and all tools or improved you can provider with Swift NIO are welcome. 😉

> That's up to the Vapor mysql driver really 🙂.
Same with PostgreSQL
My project uses postgres and I have the same issue

This fragmentation problem reminds me of a case: Several years ago I had a similar memory fragmentation problem on a projet. On this project we had a large number of objects that were allocated and deallocated intensively. Almost all objects of this project had the same size. To reduce fragmentation and make the best use of memory we had set up several heaps. Each heap being dedicated to objects with same size. Because the objects in a heap always had the same size, we could maximize the use of the memory pages with a minimum of fragmentation and a noticeable increase the performance. It's just an idea, but I wonder if this kind of approach could be interesting in terms of how Swift manages his memory. The advantage of doing so at the language level is that it would be available by default for all applications compile with Swift. Anyway, it's just an idea 😉

[weissi]

Tof (JIRA User) Right, that's getting interesting. Right now it wouldn't even be possible to use different heaps for certain {{Data}}s. Eventually the language could provide more support for custom allocators...

[weissi]

Tof (JIRA User) regarding your application with Postgres: Do you have any stats on how much peak memory should be required vs. how much peak memory is required? I'm sure the postgres driver etc could also be optimised that we get tighter in memory here. Which driver are you using btw?

[swift-ci]

Comment by Christophe Braud (JIRA)

@weissi currently the application run in a micro service environment with no debug tool, a minimalist version of ubuntu (we would have preferred Alpine Linux but Swift is not available on this micro Linux) and the application. We don't have the permission to change this environment. we have just the right to update the application. With the service administration we can see what he consumes in memory real time but without history. At startup the service takes 8 MiB. After that he takes more and more memory. As soon as the application reaches 100 MiB the service is automatically restarted. The speed at which I get to the 100 MiB depends a lot on the data flow that the service has to deal with. The traces that we put in the app allowed us to identify that as soon as we access database the service increases its memory consumption and never release it. To do well we would have to look at the vapor side to see if we can have more accurate stats.
For the moment, the service processes only a small part of the data flow and we will not be able to process more until the application will control its memory consumption.
At the driver level we use postgres and redis. But we will remove Redis as soon as possible because it consumes too much resource for what we need.
Our goal is to have services as small as possible and the most efficient possible. Swift and Vapor allow us to have good performance even with the Future and Promise layer. We hope that in a future version Swift will support the coroutines with async and await instructions base on the support of coroutines offered by LLVM it should normally greatly improve performance (exit Future and Promise!). We look forward to this version of Swift NIO 😃
For the Postgres layer, in addition to the memory issue, is not yet robust enough. That's why we follow with interest the progress on NIO Postgres. 😉
If in addition Xcode had allowed us to do remote debug on Linux on the Vapor application it would have really helped us a lot.

[weissi]

Tof (JIRA User) Alright, a couple of things here. First of all, in your case it sounds like we have no idea what the real issue is, the only similarity with the other issue is that it's memory-related, correct?

Can you change the code of your application? Maybe you could add a call to mallinfo (http://man7.org/linux/man-pages/man3/mallinfo.3.html) say every minute and log out the returned struct. That would give us the information if it's either a leak or something else.

Regarding what you write about Futures/Promises. It sounds like you believe that futures/promises add inefficiencies that async/await will resolve. That is not the case. Vapor/NIO can be very efficient because of the Futures/Promises and not despite them 🙂. NIO/Vapor use an asynchronous programming model. That allows us to not use one kernel thread per connection which increases efficiency. If you do asynchronous programming you need some forms of callbacks, they can be for example real callbacks (closures), futures/promises, or other techniques. Callbacks get you into 'callback hell' and therefore we chose futures/promises. But futures/promises are not less efficient than closures or anything, in both cases you will need to allocate and there's no way out of that. Btw, the same applies to the coroutines, for asynchronous programming, you need to allocate a 'continuation' where the program will continue when the result of a certain operation (say a write to the network) is known. When/if async/await lands, for NIO/Vapor the async/await support will likely enable us to hide the futures/promises behind the async/await support. Ie. they will still be there, the real benefit is that the user will have a nicer programming model but that's mostly syntactical.

For LLVM to be able to completely get rid of any async/await traces, the code needs to be able to run synchronously but that's impossible for network programming because you often need for the peer on the other side to do something for you to be able to make progress.

Does that make sense? We might want to move this discussion over to the forums btw 🙂