Open mmeles opened 9 months ago
コードを手元(自PC の docker )で実行したところ、実行して終了する際(ctrl+C を押下したとき)下記エラーが生じました。 これは想定される動作でしょうか?
vscode ➜ /workspaces/mem_leak $ python mem_leak.py
^CTraceback (most recent call last):
File "/usr/local/lib/python3.12/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/asyncio/base_events.py", line 664, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/workspaces/mem_leak/mem_leak.py", line 23, in main
await store.orderbook.wait()
File "/home/vscode/.local/lib/python3.12/site-packages/pybotters/store.py", line 258, in wait
await event.wait()
File "/usr/local/lib/python3.12/asyncio/locks.py", line 212, in wait
await fut
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/workspaces/mem_leak/mem_leak.py", line 27, in <module>
asyncio.run(main())
File "/usr/local/lib/python3.12/asyncio/runners.py", line 194, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/asyncio/runners.py", line 123, in run
raise KeyboardInterrupt()
KeyboardInterrupt
'# print(orderbook)'
のコメントアウト外したらちゃんとprint されていたので大丈夫そうかなと思ってます!
30分ほど実行し、ps auxf でプロセスが使用しているメモリを見てみたのですが、メモリリークはしてないように見えました。(はじめてつかうコマンドなのでよくわかってないですが...) ctrl+C で終了しても特にメモリの消費量に差はなかったです。 下記をそれぞれ記載します。
30分ほど実行した際の ps auxf の結果
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
vscode 272 0.0 0.3 597916 52232 ? Ssl 04:28 0:01 /home/vscode/.vscode-server/bin/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node -e ????const net = require('net'); ????const fs = require('fs'); ????process.stdin.pause(); ????const client = net.createConnection({ host: '127.0.0.1', port: 41521 }, () => { ?????console.error('Connection established'); ?????client.pipe(process.stdout); ?????process.stdin.pipe(client); ????}); ????client.on('close', function (hadError) { ?????console.error(hadError ? 'Remote close with error' : 'Remote close'); ?????process.exit(hadError ? 1 : 0); ????}); ????client.on('error', function (err) { ?????process.stderr.write(err && (err.stack || err.message) || String(err)); ????}); ????process.stdin.on('close', function (hadError) { ?????console.error(hadError ? 'Remote stdin close with error' : 'Remote stdin close'); ?????process.exit(hadError ? 1 : 0); ????}); ????process.on('uncaughtException', function (err) { ?????fs.writeSync(process.stderr.fd, `Uncaught Exception: ${String(err && (err.stack || err.message) || err)}\n`); ????}); ???
vscode 249 0.0 0.3 603036 54204 ? Ssl 04:28 0:01 /home/vscode/.vscode-server/bin/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node -e ????const net = require('net'); ????const fs = require('fs'); ????process.stdin.pause(); ????const client = net.createConnection({ host: '127.0.0.1', port: 41521 }, () => { ?????console.error('Connection established'); ?????client.pipe(process.stdout); ?????process.stdin.pipe(client); ????}); ????client.on('close', function (hadError) { ?????console.error(hadError ? 'Remote close with error' : 'Remote close'); ?????process.exit(hadError ? 1 : 0); ????}); ????client.on('error', function (err) { ?????process.stderr.write(err && (err.stack || err.message) || String(err)); ????}); ????process.stdin.on('close', function (hadError) { ?????console.error(hadError ? 'Remote stdin close with error' : 'Remote stdin close'); ?????process.exit(hadError ? 1 : 0); ????}); ????process.on('uncaughtException', function (err) { ?????fs.writeSync(process.stderr.fd, `Uncaught Exception: ${String(err && (err.stack || err.message) || err)}\n`); ????}); ???
vscode 178 0.0 0.0 2480 508 ? Ss 04:28 0:00 sh /home/vscode/.vscode-server/bin/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/bin/code-server --log debug --force-disable-user-env --server-data-dir /home/vscode/.vscode-server --use-host-proxy --telemetry-level all --accept-server-license-terms --host 127.0.0.1 --port 0 --connection-token-file /home/vscode/.vscode-server/data/Machine/.connection-token-1a5daa3a0231a0fbba4f14db7ec463cf99d7768e --extensions-download-dir /home/vscode/.vscode-server/extensionsCache --install-extension dbaeumer.vscode-eslint --install-extension ms-python.python --install-extension ms-python.vscode-pylance --start-server --disable-websocket-compression
vscode 187 0.4 0.7 982228 120956 ? Sl 04:28 0:12 \_ /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/out/server-main.js --log debug --force-disable-user-env --server-data-dir /home/vscode/.vscode-server --use-host-proxy --telemetry-level all --accept-server-license-terms --host 127.0.0.1 --port 0 --connection-token-file /home/vscode/.vscode-server/data/Machine/.connection-token-1a5daa3a0231a0fbba4f14db7ec463cf99d7768e --extensions-download-dir /home/vscode/.vscode-server/extensionsCache --install-extension dbaeumer.vscode-eslint --install-extension ms-python.python --install-extension ms-python.vscode-pylance --start-server --disable-websocket-compression
vscode 303 0.0 0.3 914452 55492 ? Sl 04:28 0:00 \_ /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/out/bootstrap-fork --type=fileWatcher
vscode 326 1.2 1.1 1058052 195288 ? Sl 04:28 0:35 \_ /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node --dns-result-order=ipv4first /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/out/bootstrap-fork --type=extensionHost --transformURIs --useHostProxy=true
vscode 364 0.0 0.3 602804 55388 ? Sl 04:28 0:00 | \_ /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/extensions/json-language-features/server/dist/node/jsonServerMain --node-ipc --clientProcessId=326
vscode 922 0.1 1.4 1040668 231176 ? Sl 04:29 0:05 | \_ /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node /home/vscode/.vscode-server/extensions/ms-python.vscode-pylance-2023.11.10/dist/server.bundle.js --cancellationReceive=file:2d54a24b698eca97aad63a0cdecfcb0dd0bd80de07 --node-ipc --clientProcessId=326
vscode 16333 0.0 0.3 601860 49180 ? Sl 05:13 0:00 | \_ /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node /home/vscode/.vscode-server/extensions/dbaeumer.vscode-eslint-2.4.2/server/out/eslintServer.js --node-ipc --clientProcessId=326
vscode 16341 0.1 0.4 636728 79476 ? Sl 05:13 0:00 | \_ /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/extensions/html-language-features/server/dist/node/htmlServerMain --node-ipc --clientProcessId=326
vscode 16725 0.6 0.8 697148 141248 ? Sl 05:13 0:01 | \_ /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node --max-old-space-size=3072 /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/extensions/node_modules/typescript/lib/tsserver.js --serverMode partialSemantic --useInferredProjectPerProjectRoot --disableAutomaticTypingAcquisition --cancellationPipeName /tmp/vscode-typescript1000/dd4d0ebbebf4ef817eb6/tscancellation-88b86aa67af86c58b18f.tmp* --locale en --noGetErrOnBackgroundUpdate --validateDefaultNpmLocation --useNodeIpc
vscode 16726 1.1 0.9 746108 160124 ? Sl 05:13 0:02 | \_ /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node --max-old-space-size=3072 /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/extensions/node_modules/typescript/lib/tsserver.js --useInferredProjectPerProjectRoot --enableTelemetry --cancellationPipeName /tmp/vscode-typescript1000/dd4d0ebbebf4ef817eb6/tscancellation-c33fb68b941710b3d939.tmp* --locale en --noGetErrOnBackgroundUpdate --validateDefaultNpmLocation --useNodeIpc
vscode 16750 0.0 0.3 608848 61580 ? Sl 05:13 0:00 | \_ /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/extensions/node_modules/typescript/lib/typingsInstaller.js --globalTypingsCacheLocation /home/vscode/.cache/typescript/5.2 --enableTelemetry --typesMapLocation /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/extensions/node_modules/typescript/lib/typesMap.json --validateDefaultNpmLocation
vscode 496 0.3 0.4 694752 75680 ? Sl 04:28 0:10 \_ /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/out/bootstrap-fork --type=ptyHost --logsPath /home/vscode/.vscode-server/data/logs/20231203T042819
vscode 761 0.0 0.0 12456 7312 pts/1 Ss 04:29 0:00 \_ /bin/bash --init-file /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/out/vs/workbench/contrib/terminal/browser/media/shellIntegration-bash.sh
vscode 1538 3.6 0.2 115448 33880 pts/1 Sl+ 04:30 1:42 | \_ python mem_leak.py
vscode 2950 0.0 0.0 12456 7348 pts/2 Ss 04:34 0:00 \_ /bin/bash --init-file /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/out/vs/workbench/contrib/terminal/browser/media/shellIntegration-bash.sh
vscode 3031 0.2 0.0 8952 4548 pts/2 S+ 04:35 0:05 | \_ htop
vscode 17499 0.0 0.0 12456 7408 pts/3 Ss 05:16 0:00 \_ /bin/bash --init-file /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/out/vs/workbench/contrib/terminal/browser/media/shellIntegration-bash.sh
vscode 17930 0.0 0.0 11620 3356 pts/3 R+ 05:17 0:00 \_ ps auxf
vscode 158 0.0 0.0 2480 508 ? Ss 04:28 0:00 /bin/sh
vscode 177 0.0 0.2 632156 46260 ? Sl 04:28 0:00 \_ /home/vscode/.vscode-server/bin/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node /tmp/vscode-remote-containers-server-15a0b9ac-e713-4eb8-9065-796943bca722.js
root 76 0.0 0.0 2480 564 ? Ss 04:28 0:00 /bin/sh
vscode 42 0.0 0.0 2480 1608 ? Ss 04:28 0:00 /bin/sh
root 34 0.0 0.0 2480 508 ? Ss 04:28 0:00 /bin/sh -c echo "New container started. Keep-alive process started." ; export VSCODE_REMOTE_CONTAINERS_SESSION=23ad5ef2-a037-46dc-96b6-0b05482277ea1701577694278 ; /bin/sh
root 40 0.0 0.0 2480 504 ? S 04:28 0:00 \_ /bin/sh
root 1 0.0 0.0 2480 1508 ? Ss 04:28 0:00 /bin/sh -c echo Container started trap "exit 0" 15 exec "$@" while sleep 1 & wait $!; do :; done -
root 17928 0.0 0.0 7256 556 ? S 05:17 0:00 sleep 1
ctrl+C でキャンセルした後の ps auxf の結果
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
vscode 272 0.0 0.3 597916 52232 ? Ssl 04:28 0:01 /home/vscode/.vscode-server/bin/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node -e ????const net = require('net'); ????const fs = require('fs'); ????process.stdin.pause(); ????const client = net.createConnection({ host: '127.0.0.1', port: 41521 }, () => { ?????console.error('Connection established'); ?????client.pipe(process.stdout); ?????process.stdin.pipe(client); ????}); ????client.on('close', function (hadError) { ?????console.error(hadError ? 'Remote close with error' : 'Remote close'); ?????process.exit(hadError ? 1 : 0); ????}); ????client.on('error', function (err) { ?????process.stderr.write(err && (err.stack || err.message) || String(err)); ????}); ????process.stdin.on('close', function (hadError) { ?????console.error(hadError ? 'Remote stdin close with error' : 'Remote stdin close'); ?????process.exit(hadError ? 1 : 0); ????}); ????process.on('uncaughtException', function (err) { ?????fs.writeSync(process.stderr.fd, `Uncaught Exception: ${String(err && (err.stack || err.message) || err)}\n`); ????}); ???
vscode 249 0.0 0.3 603036 54468 ? Ssl 04:28 0:01 /home/vscode/.vscode-server/bin/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node -e ????const net = require('net'); ????const fs = require('fs'); ????process.stdin.pause(); ????const client = net.createConnection({ host: '127.0.0.1', port: 41521 }, () => { ?????console.error('Connection established'); ?????client.pipe(process.stdout); ?????process.stdin.pipe(client); ????}); ????client.on('close', function (hadError) { ?????console.error(hadError ? 'Remote close with error' : 'Remote close'); ?????process.exit(hadError ? 1 : 0); ????}); ????client.on('error', function (err) { ?????process.stderr.write(err && (err.stack || err.message) || String(err)); ????}); ????process.stdin.on('close', function (hadError) { ?????console.error(hadError ? 'Remote stdin close with error' : 'Remote stdin close'); ?????process.exit(hadError ? 1 : 0); ????}); ????process.on('uncaughtException', function (err) { ?????fs.writeSync(process.stderr.fd, `Uncaught Exception: ${String(err && (err.stack || err.message) || err)}\n`); ????}); ???
vscode 178 0.0 0.0 2480 508 ? Ss 04:28 0:00 sh /home/vscode/.vscode-server/bin/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/bin/code-server --log debug --force-disable-user-env --server-data-dir /home/vscode/.vscode-server --use-host-proxy --telemetry-level all --accept-server-license-terms --host 127.0.0.1 --port 0 --connection-token-file /home/vscode/.vscode-server/data/Machine/.connection-token-1a5daa3a0231a0fbba4f14db7ec463cf99d7768e --extensions-download-dir /home/vscode/.vscode-server/extensionsCache --install-extension dbaeumer.vscode-eslint --install-extension ms-python.python --install-extension ms-python.vscode-pylance --start-server --disable-websocket-compression
vscode 187 0.4 0.7 982228 121560 ? Sl 04:28 0:12 \_ /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/out/server-main.js --log debug --force-disable-user-env --server-data-dir /home/vscode/.vscode-server --use-host-proxy --telemetry-level all --accept-server-license-terms --host 127.0.0.1 --port 0 --connection-token-file /home/vscode/.vscode-server/data/Machine/.connection-token-1a5daa3a0231a0fbba4f14db7ec463cf99d7768e --extensions-download-dir /home/vscode/.vscode-server/extensionsCache --install-extension dbaeumer.vscode-eslint --install-extension ms-python.python --install-extension ms-python.vscode-pylance --start-server --disable-websocket-compression
vscode 303 0.0 0.3 914452 55492 ? Sl 04:28 0:00 \_ /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/out/bootstrap-fork --type=fileWatcher
vscode 326 1.2 1.2 1058052 196112 ? Sl 04:28 0:36 \_ /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node --dns-result-order=ipv4first /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/out/bootstrap-fork --type=extensionHost --transformURIs --useHostProxy=true
vscode 364 0.0 0.3 602804 55388 ? Sl 04:28 0:00 | \_ /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/extensions/json-language-features/server/dist/node/jsonServerMain --node-ipc --clientProcessId=326
vscode 922 0.1 1.4 1040668 231176 ? Sl 04:29 0:05 | \_ /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node /home/vscode/.vscode-server/extensions/ms-python.vscode-pylance-2023.11.10/dist/server.bundle.js --cancellationReceive=file:2d54a24b698eca97aad63a0cdecfcb0dd0bd80de07 --node-ipc --clientProcessId=326
vscode 16333 0.0 0.3 601860 49180 ? Sl 05:13 0:00 | \_ /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node /home/vscode/.vscode-server/extensions/dbaeumer.vscode-eslint-2.4.2/server/out/eslintServer.js --node-ipc --clientProcessId=326
vscode 16341 0.1 0.4 636728 79476 ? Sl 05:13 0:00 | \_ /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/extensions/html-language-features/server/dist/node/htmlServerMain --node-ipc --clientProcessId=326
vscode 16725 0.5 0.8 697148 141248 ? Sl 05:13 0:01 | \_ /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node --max-old-space-size=3072 /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/extensions/node_modules/typescript/lib/tsserver.js --serverMode partialSemantic --useInferredProjectPerProjectRoot --disableAutomaticTypingAcquisition --cancellationPipeName /tmp/vscode-typescript1000/dd4d0ebbebf4ef817eb6/tscancellation-88b86aa67af86c58b18f.tmp* --locale en --noGetErrOnBackgroundUpdate --validateDefaultNpmLocation --useNodeIpc
vscode 16726 1.0 0.9 746108 160124 ? Sl 05:13 0:02 | \_ /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node --max-old-space-size=3072 /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/extensions/node_modules/typescript/lib/tsserver.js --useInferredProjectPerProjectRoot --enableTelemetry --cancellationPipeName /tmp/vscode-typescript1000/dd4d0ebbebf4ef817eb6/tscancellation-c33fb68b941710b3d939.tmp* --locale en --noGetErrOnBackgroundUpdate --validateDefaultNpmLocation --useNodeIpc
vscode 16750 0.0 0.3 608848 61580 ? Sl 05:13 0:00 | \_ /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/extensions/node_modules/typescript/lib/typingsInstaller.js --globalTypingsCacheLocation /home/vscode/.cache/typescript/5.2 --enableTelemetry --typesMapLocation /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/extensions/node_modules/typescript/lib/typesMap.json --validateDefaultNpmLocation
vscode 496 0.3 0.4 694752 75604 ? Sl 04:28 0:10 \_ /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/out/bootstrap-fork --type=ptyHost --logsPath /home/vscode/.vscode-server/data/logs/20231203T042819
vscode 761 0.0 0.0 12456 7312 pts/1 Ss+ 04:29 0:00 \_ /bin/bash --init-file /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/out/vs/workbench/contrib/terminal/browser/media/shellIntegration-bash.sh
vscode 2950 0.0 0.0 12456 7348 pts/2 Ss 04:34 0:00 \_ /bin/bash --init-file /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/out/vs/workbench/contrib/terminal/browser/media/shellIntegration-bash.sh
vscode 3031 0.2 0.0 8952 4548 pts/2 S+ 04:35 0:05 | \_ htop
vscode 17499 0.0 0.0 12456 7408 pts/3 Ss 05:16 0:00 \_ /bin/bash --init-file /vscode/vscode-server/bin/linux-x64/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/out/vs/workbench/contrib/terminal/browser/media/shellIntegration-bash.sh
vscode 18191 0.0 0.0 11620 3388 pts/3 R+ 05:17 0:00 \_ ps auxf
vscode 158 0.0 0.0 2480 508 ? Ss 04:28 0:00 /bin/sh
vscode 177 0.0 0.2 632156 46260 ? Sl 04:28 0:00 \_ /home/vscode/.vscode-server/bin/1a5daa3a0231a0fbba4f14db7ec463cf99d7768e/node /tmp/vscode-remote-containers-server-15a0b9ac-e713-4eb8-9065-796943bca722.js
root 76 0.0 0.0 2480 564 ? Ss 04:28 0:00 /bin/sh
vscode 42 0.0 0.0 2480 1608 ? Ss 04:28 0:00 /bin/sh
root 34 0.0 0.0 2480 508 ? Ss 04:28 0:00 /bin/sh -c echo "New container started. Keep-alive process started." ; export VSCODE_REMOTE_CONTAINERS_SESSION=23ad5ef2-a037-46dc-96b6-0b05482277ea1701577694278 ; /bin/sh
root 40 0.0 0.0 2480 504 ? S 04:28 0:00 \_ /bin/sh
root 1 0.0 0.0 2480 1508 ? Ss 04:28 0:00 /bin/sh -c echo Container started trap "exit 0" 15 exec "$@" while sleep 1 & wait $!; do :; done -
root 18189 0.0 0.0 7256 500 ? S 05:17 0:00 sleep 1
下記が考えられると思います。
なので、
などが気になります。 もしかしたら docker を使ったのが微妙かもしれない...
memory_profilerで各行ごとに調べてみたものになります。ループ回数で違っていました。ループ数100のほう
Filename: test_program.py
5 30.6 MiB 30.6 MiB 1 @profile
6 async def main():
7 30.6 MiB 0.0 MiB 1 async with pybotters.Client() as client:
8 30.6 MiB 0.0 MiB 1 store = pybotters.BybitV5DataStore()
9 30.6 MiB 0.0 MiB 1 ws = await client.ws_connect(
10 30.6 MiB 0.0 MiB 1 'wss://stream.bybit.com/v5/public/linear',
11 send_json={
12 30.6 MiB 0.0 MiB 1 'op': 'subscribe',
13 30.6 MiB 0.0 MiB 1 'args': ['orderbook.50.BTCUSDT'],
14 },
15 32.8 MiB 2.2 MiB 2 hdlr_json=store.onmessage,
16 )
17 33.0 MiB 0.0 MiB 2 while not all([
18 33.0 MiB 0.0 MiB 2 len(store.orderbook),
19 ]):
20 33.0 MiB 0.2 MiB 2 await store.wait()
21
22 33.0 MiB 0.0 MiB 102 for _ in range(100):
23 33.0 MiB 0.0 MiB 200 await store.orderbook.wait()
24 33.0 MiB 0.0 MiB 100 orderbook = store.orderbook.sorted({'symbol': 'BTCUSDT'})
ループ数1000にした方。23行目が異なる。 記述されてない上4行は以下のモジュールの読み込み+改行 import asyncio import pybotters from memory_profiler import profile
Filename: test_program.py
5 30.8 MiB 30.8 MiB 1 @profile
6 async def main():
7 30.8 MiB 0.0 MiB 1 async with pybotters.Client() as client:
8 30.8 MiB 0.0 MiB 1 store = pybotters.BybitV5DataStore()
9 30.8 MiB 0.0 MiB 1 ws = await client.ws_connect(
10 30.8 MiB 0.0 MiB 1 'wss://stream.bybit.com/v5/public/linear',
11 send_json={
12 30.8 MiB 0.0 MiB 1 'op': 'subscribe',
13 30.8 MiB 0.0 MiB 1 'args': ['orderbook.50.BTCUSDT'],
14 },
15 32.8 MiB 2.0 MiB 2 hdlr_json=store.onmessage,
16 )
17 33.1 MiB 0.0 MiB 2 while not all([
18 33.1 MiB 0.0 MiB 2 len(store.orderbook),
19 ]):
20 33.1 MiB 0.2 MiB 2 await store.wait()
21
22 33.4 MiB 0.0 MiB 1002 for _ in range(1000):
23 33.4 MiB 0.3 MiB 2000 await store.orderbook.wait()
24 33.4 MiB 0.0 MiB 1000 orderbook = store.orderbook.sorted({'symbol': 'BTCUSDT'})
ループ数変えてやってみたところ、23行目に記載されたメモリは下記の通りでした
100:0.2
1000:0.8
10000:0.7
100000:2.0
(MiB)
確かにメモリの消費量が増える傾向ありそうです。 謎も多いです。
次は500000で回してみます(時間かかりすぎて諦めるかも…)
環境はひとまず下記でやっています。
500000は何回やってもファイルが残らないので、200000で実行したらよくわからない結果になってました... とりあえず結果だけ貼ります。時間があるときにまた調べます。
Line # Mem usage Increment Occurrences Line Contents
=============================================================
5 30.6 MiB 30.6 MiB 1 @profile
6 async def main():
7 30.8 MiB 0.2 MiB 1 async with pybotters.Client() as client:
8 30.8 MiB 0.0 MiB 1 store = pybotters.BybitV5DataStore()
9 30.8 MiB 0.0 MiB 1 ws = await http://client.ws_connect(
10 30.8 MiB 0.0 MiB 1 'wss://stream.bybit.com/v5/public/linear',
11 send_json={
12 30.8 MiB 0.0 MiB 1 'op': 'subscribe',
13 30.8 MiB 0.0 MiB 1 'args': ['orderbook.50.BTCUSDT'],
14 },
15 32.7 MiB 1.9 MiB 2 hdlr_json=store.onmessage,
16 )
17 32.7 MiB 0.0 MiB 2 while not all([
18 32.7 MiB 0.0 MiB 2 len(store.orderbook),
19 ]):
20 32.7 MiB 0.0 MiB 2 await store.wait()
21 # 以下取得したordebookを元に発注を繰り返す。
22 37.7 MiB -22580.4 MiB 200002 for _ in range (200000):
23 37.7 MiB -45154.2 MiB 400000 await store.orderbook.wait()
24 37.7 MiB -22579.9 MiB 200000 orderbook = store.orderbook.sorted({'symbol': 'BTCUSDT'})
ありがと~、なんだこれはw
きちんと記録取れていないですが、同じループ回数でもメモリ消費量が変わる場合がある気がします… 調査に役立つかもしれないので残しておきます。
コード追ってみたのですが、メモリリークの箇所はわかりませんでした...
わかったところだけ下記に記載します
下記のように bybit_v5 .py に sleep を入れると事象が再現しやすい(ループ数 3 回とかでもメモリ使ってくれます。時間は環境によって変えたほうが良さそうです) def _onmessage(self, msg: Item, topic_ext: list[str]) -> None: time.sleep(0.04) operation = {"delete": [], "update": [], "insert": []}
メモリの消費は下記箇所の★の行で起こっている(loop のたびに呼ばれて、その累積値が最終的な増分値になります) 。 ここから先は await event.wait() の時にどの処理が動いているのか追いきれませんでした... async def wait(self) -> list[Item]: event = asyncio.Event() ret = [] self._events[event] = ret await event.wait() ★★ del self._events[event] return ret
@profile をライブラリ内に色々と仕込んで動かしたところ、下記が怪しそうな箇所の一つでした。ただ、他にもリークしてそうなところがあり数値は合わなかったので正直参考程度です...どんな処理をしているのかは追えていません。 Filename: /usr/local/lib/python3.7/site-packages/pybotters/models/bybit_v5.py
68 38.7 MiB 38.7 MiB 1 @profile
69 def _onmessage(self, msg: Item, ws: ClientWebSocketResponse) -> None:
70 38.7 MiB 0.0 MiB 1 if "success" in msg:
71 if not msg["success"]:
72 logger.warning(msg)
73
74 38.7 MiB 0.0 MiB 1 if "topic" in msg:
75 38.7 MiB 0.0 MiB 1 dot_topic: str = msg["topic"]
76 39.3 MiB 0.5 MiB 1 topic, *topic_ext = dot_topic.split(".") ★★
77
78 39.3 MiB 0.0 MiB 1 if topic in self:
79 39.3 MiB 0.0 MiB 1 getattr(self[topic], "_onmessage")(msg, topic_ext)
68 35.8 MiB 35.8 MiB 1 @profile
69 def _onmessage(self, msg: Item, ws: ClientWebSocketResponse) -> None:
70 35.8 MiB 0.0 MiB 1 if "success" in msg:
71 if not msg["success"]:
72 logger.warning(msg)
73
74 35.8 MiB 0.0 MiB 1 if "topic" in msg:
75 35.8 MiB 0.0 MiB 1 dot_topic: str = msg["topic"]
76 35.8 MiB 0.0 MiB 1 topic, *topic_ext = dot_topic.split(".")
77
78 35.8 MiB 0.0 MiB 1 if topic in self:
79 36.2 MiB 0.3 MiB 1 getattr(self[topic], "_onmessage")(msg, topic_ext) ★★
いつか時間があればさらに調査したいですが、当面は時間を取るのが難しそうです…
websocket を使用するようになってからエラーが生じる