search

ydb-platform / ydb

YDB is an open source Distributed SQL Database that combines high availability and scalability with strong consistency and ACID transactions
https://ydb.tech
Apache License 2.0
4.02k stars 583 forks source link

Improve csv import, steps 1 & 2 #11679

Closed pnv1 closed 3 days ago

pnv1 commented 1 week ago

Changelog entry

Changelog category

Issue: https://github.com/ydb-platform/ydb/issues/11678

github-actions[bot] commented 1 week ago

:white_circle: 2024-11-15 19:42:43 UTC Pre-commit check linux-x86_64-relwithdebinfo for bf6067811dd1b36fb14c696a465990588e05974d has started. :white_circle: 2024-11-15 19:42:55 UTC Artifacts will be uploaded here :white_circle: 2024-11-15 19:45:54 UTC ya make is running... :yellow_circle: 2024-11-15 20:42:17 UTC Some tests failed, follow the links below. Going to retry failed tests...

[Test history](https://nebius.testmo.net/automation/runs/view/79420) | [Ya make output](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11862437839/ya-x86-64/ya_make_output.txt) | [Test bloat](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11862437839/ya-x86-64/try_1/test_bloat/tree_map.html) | TESTS | PASSED | ERRORS | FAILED | SKIPPED | MUTED[?](https://github.com/ydb-platform/ydb/tree/main/.github/config/muted_ya.txt "All mute rules are defined here") | | ---: | ---: | ---: | ---: | ---: | ---: | | [40286](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11862437839/ya-x86-64/try_1/ya-test.html) | [34576](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11862437839/ya-x86-64/try_1/ya-test.html#PASS) | 0 | [12](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11862437839/ya-x86-64/try_1/ya-test.html#FAIL) | [5580](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11862437839/ya-x86-64/try_1/ya-test.html#SKIP) | [118](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11862437839/ya-x86-64/try_1/ya-test.html#MUTE) |

:white_circle: 2024-11-15 20:45:47 UTC ya make is running... (failed tests rerun, try 2) :yellow_circle: 2024-11-15 20:57:25 UTC Some tests failed, follow the links below. Going to retry failed tests...

[Test history](https://nebius.testmo.net/automation/runs/view/79430) | [Ya make output](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11862437839/ya-x86-64/ya_make_output.txt) | [Test bloat](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11862437839/ya-x86-64/try_1/test_bloat/tree_map.html) | [Test bloat](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11862437839/ya-x86-64/try_2/test_bloat/tree_map.html) | TESTS | PASSED | ERRORS | FAILED | SKIPPED | MUTED[?](https://github.com/ydb-platform/ydb/tree/main/.github/config/muted_ya.txt "All mute rules are defined here") | | ---: | ---: | ---: | ---: | ---: | ---: | | [150 (only retried tests)](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11862437839/ya-x86-64/try_2/ya-test.html) | [35](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11862437839/ya-x86-64/try_2/ya-test.html#PASS) | 0 | [8](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11862437839/ya-x86-64/try_2/ya-test.html#FAIL) | 0 | [107](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11862437839/ya-x86-64/try_2/ya-test.html#MUTE) |

:white_circle: 2024-11-15 20:57:33 UTC ya make is running... (failed tests rerun, try 3) :red_circle: 2024-11-15 21:08:50 UTC Some tests failed, follow the links below.

Test history Ya make output Test bloat Test bloat Test bloat TESTS PASSED ERRORS FAILED SKIPPED MUTED?
115 (only retried tests) 0 0 8 0 107
:green_circle: 2024-11-15 21:08:57 UTC Build successful. :green_circle: 2024-11-15 21:09:17 UTC ydbd size 2.5 GiB changed* by 0 Bytes, which is <= 0 Bytes vs main: OK ydbd size dash main: 8f7b4e411bb96a98810f27c17398aeaf706dde6d merge: bf6067811dd1b36fb14c696a465990588e05974d diff diff %
ydbd size 2 682 835 888 Bytes 2 682 835 888 Bytes 0 Bytes 0.000%
ydbd stripped size 480 948 144 Bytes 480 948 144 Bytes 0 Bytes 0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

github-actions[bot] commented 1 week ago

:white_circle: 2024-11-15 19:42:56 UTC Pre-commit check linux-x86_64-release-asan for bf6067811dd1b36fb14c696a465990588e05974d has started. :white_circle: 2024-11-15 19:43:07 UTC Artifacts will be uploaded here :white_circle: 2024-11-15 19:46:07 UTC ya make is running... :yellow_circle: 2024-11-15 20:49:14 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history Ya make output Test bloat TESTS PASSED ERRORS FAILED SKIPPED MUTED?
9087 9013 0 22 5 47
:green_circle: 2024-11-15 20:50:01 UTC Build successful. :yellow_circle: 2024-11-15 20:50:28 UTC ydbd size 4.9 GiB changed* by +300.6 KiB, which is >= 100.0 KiB vs main: Warning ydbd size dash main: 78eb45ead832afe0e2c807f13127bede90df264c merge: bf6067811dd1b36fb14c696a465990588e05974d diff diff %
ydbd size 5 257 040 160 Bytes 5 257 348 000 Bytes +300.6 KiB +0.006%
ydbd stripped size 1 350 528 400 Bytes 1 350 603 792 Bytes +73.6 KiB +0.006%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

github-actions[bot] commented 5 days ago

:white_circle: 2024-11-17 09:39:07 UTC Pre-commit check linux-x86_64-release-asan for 852b8faae43456297a0960532a36c94927a21d0f has started. :white_circle: 2024-11-17 09:39:19 UTC Artifacts will be uploaded here :white_circle: 2024-11-17 09:42:35 UTC ya make is running... :red_circle: 2024-11-17 09:48:09 UTC Build failed, see the logs. Also see fail summary

github-actions[bot] commented 5 days ago

:white_circle: 2024-11-17 09:39:29 UTC Pre-commit check linux-x86_64-relwithdebinfo for 852b8faae43456297a0960532a36c94927a21d0f has started. :white_circle: 2024-11-17 09:39:40 UTC Artifacts will be uploaded here :white_circle: 2024-11-17 09:43:05 UTC ya make is running... :red_circle: 2024-11-17 09:45:24 UTC Build failed, see the logs. Also see fail summary

github-actions[bot] commented 5 days ago

:white_circle: 2024-11-17 09:57:55 UTC Pre-commit check linux-x86_64-relwithdebinfo for 8876750c93c1b2f614a86d93f9aecf83b102e1d7 has started. :white_circle: 2024-11-17 09:58:05 UTC Artifacts will be uploaded here :white_circle: 2024-11-17 10:01:27 UTC ya make is running... :yellow_circle: 2024-11-17 11:32:50 UTC Some tests failed, follow the links below. Going to retry failed tests...

[Test history](https://nebius.testmo.net/automation/runs/view/79530) | [Ya make output](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11877867719/ya-x86-64/ya_make_output.txt) | [Test bloat](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11877867719/ya-x86-64/try_1/test_bloat/tree_map.html) | TESTS | PASSED | ERRORS | FAILED | SKIPPED | MUTED[?](https://github.com/ydb-platform/ydb/tree/main/.github/config/muted_ya.txt "All mute rules are defined here") | | ---: | ---: | ---: | ---: | ---: | ---: | | [75307](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11877867719/ya-x86-64/try_1/ya-test.html) | [61194](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11877867719/ya-x86-64/try_1/ya-test.html#PASS) | 0 | [11](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11877867719/ya-x86-64/try_1/ya-test.html#FAIL) | [13987](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11877867719/ya-x86-64/try_1/ya-test.html#SKIP) | [115](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11877867719/ya-x86-64/try_1/ya-test.html#MUTE) |

:white_circle: 2024-11-17 11:39:52 UTC ya make is running... (failed tests rerun, try 2) :yellow_circle: 2024-11-17 11:51:35 UTC Some tests failed, follow the links below. Going to retry failed tests...

[Test history](https://nebius.testmo.net/automation/runs/view/79538) | [Ya make output](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11877867719/ya-x86-64/ya_make_output.txt) | [Test bloat](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11877867719/ya-x86-64/try_1/test_bloat/tree_map.html) | [Test bloat](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11877867719/ya-x86-64/try_2/test_bloat/tree_map.html) | TESTS | PASSED | ERRORS | FAILED | SKIPPED | MUTED[?](https://github.com/ydb-platform/ydb/tree/main/.github/config/muted_ya.txt "All mute rules are defined here") | | ---: | ---: | ---: | ---: | ---: | ---: | | [126 (only retried tests)](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11877867719/ya-x86-64/try_2/ya-test.html) | [9](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11877867719/ya-x86-64/try_2/ya-test.html#PASS) | 0 | [8](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11877867719/ya-x86-64/try_2/ya-test.html#FAIL) | 0 | [109](https://storage.yandexcloud.net/ydb-gh-logs/ydb-platform/ydb/PR-check/11877867719/ya-x86-64/try_2/ya-test.html#MUTE) |

:white_circle: 2024-11-17 11:51:43 UTC ya make is running... (failed tests rerun, try 3) :red_circle: 2024-11-17 12:03:19 UTC Some tests failed, follow the links below.

Test history Ya make output Test bloat Test bloat Test bloat TESTS PASSED ERRORS FAILED SKIPPED MUTED?
117 (only retried tests) 2 0 8 0 107
:green_circle: 2024-11-17 12:03:26 UTC Build successful. :green_circle: 2024-11-17 12:03:48 UTC ydbd size 2.5 GiB changed* by +448 Bytes, which is < 100.0 KiB vs main: OK ydbd size dash main: 16b2be5def46f7363f5dc84ff982996219de6fd0 merge: 8876750c93c1b2f614a86d93f9aecf83b102e1d7 diff diff %
ydbd size 2 683 042 192 Bytes 2 683 042 640 Bytes +448 Bytes +0.000%
ydbd stripped size 480 973 488 Bytes 480 973 488 Bytes 0 Bytes 0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

github-actions[bot] commented 5 days ago

:white_circle: 2024-11-17 09:58:09 UTC Pre-commit check linux-x86_64-release-asan for 8876750c93c1b2f614a86d93f9aecf83b102e1d7 has started. :white_circle: 2024-11-17 09:58:19 UTC Artifacts will be uploaded here :white_circle: 2024-11-17 10:01:36 UTC ya make is running... :yellow_circle: 2024-11-17 11:28:20 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history Ya make output Test bloat TESTS PASSED ERRORS FAILED SKIPPED MUTED?
10614 10499 0 24 44 47
:green_circle: 2024-11-17 11:29:15 UTC Build successful. :green_circle: 2024-11-17 11:29:41 UTC ydbd size 4.9 GiB changed* by +1.1 KiB, which is < 100.0 KiB vs main: OK ydbd size dash main: 16b2be5def46f7363f5dc84ff982996219de6fd0 merge: 8876750c93c1b2f614a86d93f9aecf83b102e1d7 diff diff %
ydbd size 5 257 825 688 Bytes 5 257 826 840 Bytes +1.1 KiB +0.000%
ydbd stripped size 1 350 702 320 Bytes 1 350 702 512 Bytes +192 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

github-actions[bot] commented 4 days ago

:white_circle: 2024-11-18 08:02:57 UTC Pre-commit check linux-x86_64-release-asan for f17e041d715b286bcc58b9acbc6766b6946c7dd4 has started. :white_circle: 2024-11-18 08:03:08 UTC Artifacts will be uploaded here :white_circle: 2024-11-18 08:06:37 UTC ya make is running... :yellow_circle: 2024-11-18 09:39:37 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history Ya make output Test bloat TESTS PASSED ERRORS FAILED SKIPPED MUTED?
10616 10540 0 20 7 49
:green_circle: 2024-11-18 09:40:31 UTC Build successful. :green_circle: 2024-11-18 09:41:03 UTC ydbd size 4.9 GiB changed* by +1.1 KiB, which is < 100.0 KiB vs main: OK ydbd size dash main: 564ee81a6927ad3f2e72a268c383dcad4a97e105 merge: f17e041d715b286bcc58b9acbc6766b6946c7dd4 diff diff %
ydbd size 5 258 842 352 Bytes 5 258 843 496 Bytes +1.1 KiB +0.000%
ydbd stripped size 1 350 965 264 Bytes 1 350 965 456 Bytes +192 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

github-actions[bot] commented 4 days ago

:white_circle: 2024-11-18 08:04:12 UTC Pre-commit check linux-x86_64-relwithdebinfo for f17e041d715b286bcc58b9acbc6766b6946c7dd4 has started. :white_circle: 2024-11-18 08:04:25 UTC Artifacts will be uploaded here :white_circle: 2024-11-18 08:07:41 UTC ya make is running... :green_circle: 2024-11-18 09:29:56 UTC Tests successful.

Test history Ya make output Test bloat TESTS PASSED ERRORS FAILED SKIPPED MUTED?
75307 61210 0 0 13984 113
:green_circle: 2024-11-18 09:36:59 UTC Build successful. :yellow_circle: 2024-11-18 09:37:17 UTC ydbd size 2.5 GiB changed* by +277.5 KiB, which is >= 100.0 KiB vs main: Warning ydbd size dash main: 3d691abfb093878b7ec448d037fd462c13422876 merge: f17e041d715b286bcc58b9acbc6766b6946c7dd4 diff diff %
ydbd size 2 683 092 736 Bytes 2 683 376 928 Bytes +277.5 KiB +0.011%
ydbd stripped size 480 985 712 Bytes 481 038 256 Bytes +51.3 KiB +0.011%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

github-actions[bot] commented 4 days ago

:white_circle: 2024-11-18 17:17:33 UTC Pre-commit check linux-x86_64-release-asan for 2c78011e94e787f11c83f20d6f70c921fa697bc0 has started. :white_circle: 2024-11-18 17:17:44 UTC Artifacts will be uploaded here :white_circle: 2024-11-18 17:20:44 UTC ya make is running... :black_circle: 2024-11-18 17:35:48 UTC Check cancelled

github-actions[bot] commented 4 days ago

:white_circle: 2024-11-18 17:18:36 UTC Pre-commit check linux-x86_64-relwithdebinfo for 2c78011e94e787f11c83f20d6f70c921fa697bc0 has started. :white_circle: 2024-11-18 17:18:47 UTC Artifacts will be uploaded here :white_circle: 2024-11-18 17:21:46 UTC ya make is running... :black_circle: 2024-11-18 17:35:44 UTC Check cancelled

github-actions[bot] commented 4 days ago

:white_circle: 2024-11-18 17:37:32 UTC Pre-commit check linux-x86_64-release-asan for 6899e210b117f4bf6e79cd70a8ec437a4725ce50 has started. :white_circle: 2024-11-18 17:39:05 UTC Artifacts will be uploaded here :white_circle: 2024-11-18 17:42:34 UTC ya make is running... :red_circle: 2024-11-18 17:44:51 UTC Build failed, see the logs. Also see fail summary

github-actions[bot] commented 4 days ago

:white_circle: 2024-11-18 17:39:35 UTC Pre-commit check linux-x86_64-relwithdebinfo for 6899e210b117f4bf6e79cd70a8ec437a4725ce50 has started. :white_circle: 2024-11-18 17:39:46 UTC Artifacts will be uploaded here :white_circle: 2024-11-18 17:42:46 UTC ya make is running... :red_circle: 2024-11-18 17:45:38 UTC Build failed, see the logs. Also see fail summary

github-actions[bot] commented 4 days ago

:white_circle: 2024-11-18 19:06:40 UTC Pre-commit check linux-x86_64-release-asan for 080f27a1bb324f41dff224a471062b4831501862 has started. :white_circle: 2024-11-18 19:07:15 UTC Artifacts will be uploaded here :white_circle: 2024-11-18 19:10:47 UTC ya make is running... :yellow_circle: 2024-11-18 20:10:22 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history Ya make output Test bloat TESTS PASSED ERRORS FAILED SKIPPED MUTED?
9099 9026 0 19 8 46
:green_circle: 2024-11-18 20:11:14 UTC Build successful. :green_circle: 2024-11-18 20:11:40 UTC ydbd size 4.9 GiB changed* by -96 Bytes, which is <= 0 Bytes vs main: OK ydbd size dash main: fc93800af71ca2799d0847d1d30431074b47a38e merge: 080f27a1bb324f41dff224a471062b4831501862 diff diff %
ydbd size 5 259 636 264 Bytes 5 259 636 168 Bytes -96 Bytes -0.000%
ydbd stripped size 1 351 086 768 Bytes 1 351 086 704 Bytes -64 Bytes -0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

github-actions[bot] commented 4 days ago

:white_circle: 2024-11-19 15:01:13 UTC Pre-commit check linux-x86_64-relwithdebinfo for 080f27a1bb324f41dff224a471062b4831501862 has started. :white_circle: 2024-11-19 15:01:45 UTC Artifacts will be uploaded here :white_circle: 2024-11-19 15:05:24 UTC ya make is running... :green_circle: 2024-11-19 16:00:51 UTC Tests successful.

Test history Ya make output Test bloat TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40302 34624 0 0 5558 120
:green_circle: 2024-11-19 16:04:11 UTC Build successful. :green_circle: 2024-11-19 16:04:28 UTC ydbd size 2.5 GiB changed* by 0 Bytes, which is <= 0 Bytes vs main: OK ydbd size dash main: fc93800af71ca2799d0847d1d30431074b47a38e merge: 080f27a1bb324f41dff224a471062b4831501862 diff diff %
ydbd size 2 683 723 592 Bytes 2 683 723 592 Bytes 0 Bytes 0.000%
ydbd stripped size 481 086 000 Bytes 481 086 000 Bytes 0 Bytes 0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation