search

Mooncake-Labs / pg_mooncake

Iceberg/Delta Columnstore Table in Postgres
http://mooncake.dev
MIT License
228 stars 12 forks source link

how to make ceph radosgw as storage? #37

Open bpint opened 3 days ago

bpint commented 3 days ago

What happens?

i tried to use ceph radosgw as an s3 storage for pg_mooncake but failed. did i do something wrong?

To Reproduce

first, i built pg_mooncake from latest git with this fix, for postgresql 16.4 on RHEL8-like system.

the ceph 14.2.22 rgw was configured on http://192.168.2.201 and the bucket name is 'test'. i tried,

CREATE EXTENSION pg_mooncake;

SELECT mooncake.create_secret('test', 'S3', 'J99FFEUCNV428W9DXS5M', '0nGNMIACzFOpRgD1XuqbmtOhv7V3mQWIPT71eAOt', '{"ENDPOINT":"192.168.2.201", "USE_SSL": 0}');
SET mooncake.default_bucket = 's3://test';

CREATE TABLE pg_mooncake_test(id BIGINT, log TEXT, ts TIMESTAMP) USING columnstore;
INSERT INTO pg_mooncake_test VALUES (1, 'test', '2024-01-01 08:00:00');
SELECT * FROM pg_mooncake_test;
DROP TABLE pg_mooncake_test;

CREATE TABLE succeeded, while INSERT INTO failed with,

ERROR:  (PGDuckDB/ExecuteQuery) IO Error: Could not establish connection error for HTTP HEAD to 'http://test.192.168.2.201/mooncake_postgres_pg_mooncake_test_16419/08b0028e-cf3d-4942-9225-2a90c4880e1b.parquet'

add host name "test.192.168.2.201" to /etc/hosts was not a workaround, since ceph expected something like http://192.168.2.201/<bucket-name>/<path>/, but pg_mooncake sent http://<bucket-name>.192.168.2.201/<path>/ instead. it seams nginx reverse proxy may fix this but i did not succeeded on it.

why pg_mooncake talked with rgw well during CREATE TABLE but not INSERT INTO?

ceph rgw log,

2024-11-19 09:56:51.908 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-19 09:56:51.924 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.016s ======
2024-11-19 09:56:51.925 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [19/Nov/2024:09:56:51 +0800] "GET /test?list-type=2&prefix=mooncake_postgres_pg_mooncake_test_16419%2F_delta_log%2F HTTP/1.1" 200 480 - object_store/0.10.2
2024-11-19 09:56:51.997 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-19 09:56:52.701 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.704s ======
2024-11-19 09:56:52.701 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [19/Nov/2024:09:56:51 +0800] "PUT /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/_commit_35a2dc44-909a-4910-a9bc-8bd6015e1c10.json.tmp HTTP/1.1" 200 240 - object_store/0.10.2
2024-11-19 09:56:52.747 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-19 09:56:53.024 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.277s ======
2024-11-19 09:56:53.024 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [19/Nov/2024:09:56:51 +0800] "PUT /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/00000000000000000000.json HTTP/1.1" 200 420 - object_store/0.10.2
2024-11-19 09:56:53.127 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-19 09:56:53.270 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=204 latency=0.143s ======
2024-11-19 09:56:53.270 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [19/Nov/2024:09:56:53 +0800] "DELETE /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/_commit_35a2dc44-909a-4910-a9bc-8bd6015e1c10.json.tmp HTTP/1.1" 204 131 - object_store/0.10.2
2024-11-19 09:56:53.408 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-19 09:56:53.411 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=404 latency=0.003s ======
2024-11-19 09:56:53.411 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [19/Nov/2024:09:56:53 +0800] "GET /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/_last_checkpoint HTTP/1.1" 404 415 - object_store/0.10.2
2024-11-19 09:56:53.528 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-19 09:56:53.534 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.006s ======
2024-11-19 09:56:53.534 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [19/Nov/2024:09:56:53 +0800] "GET /test?list-type=2&prefix=mooncake_postgres_pg_mooncake_test_16419%2F_delta_log%2F&start-after=mooncake_postgres_pg_mooncake_test_16419%2F_delta_log%2F00000000000000000000 HTTP/1.1" 200 920 - object_store/0.10.2
2024-11-19 09:56:53.656 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-19 09:56:53.661 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.005s ======
2024-11-19 09:56:53.661 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [19/Nov/2024:09:56:53 +0800] "GET /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/00000000000000000000.json HTTP/1.1" 200 2099 - object_store/0.10.2
2024-11-19 09:56:53.691 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-19 09:56:53.694 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.003s ======
2024-11-19 09:56:53.695 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [19/Nov/2024:09:56:53 +0800] "GET /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/00000000000000000000.json HTTP/1.1" 200 2099 - object_store/0.10.2
2024-11-19 09:56:53.726 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-19 09:56:53.728 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=404 latency=0.002s ======
2024-11-19 09:56:53.728 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [19/Nov/2024:09:56:53 +0800] "GET /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/_last_checkpoint HTTP/1.1" 404 415 - object_store/0.10.2
2024-11-19 09:56:53.840 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-19 09:56:53.846 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.006s ======
2024-11-19 09:56:53.846 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [19/Nov/2024:09:56:53 +0800] "GET /test?list-type=2&prefix=mooncake_postgres_pg_mooncake_test_16419%2F_delta_log%2F&start-after=mooncake_postgres_pg_mooncake_test_16419%2F_delta_log%2F00000000000000000000 HTTP/1.1" 200 920 - object_store/0.10.2
2024-11-19 09:56:53.955 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-19 09:56:53.958 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.003s ======
2024-11-19 09:56:53.959 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [19/Nov/2024:09:56:53 +0800] "GET /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/00000000000000000000.json HTTP/1.1" 200 2099 - object_store/0.10.2
2024-11-19 09:56:53.989 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-19 09:56:53.993 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.004s ======
2024-11-19 09:56:53.993 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [19/Nov/2024:09:56:53 +0800] "GET /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/00000000000000000000.json HTTP/1.1" 200 2099 - object_store/0.10.2
2024-11-19 09:57:26.724 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-19 09:57:26.727 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=404 latency=0.003s ======
2024-11-19 09:57:26.727 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [19/Nov/2024:09:57:26 +0800] "HEAD /mooncake_postgres_pg_mooncake_test_16419/6834f954-a236-4605-ad73-d6cece3c19c4.parquet HTTP/1.1" 404 204 - cpp-httplib/0.14.3
2024-11-19 09:57:26.895 7fc1d8e54700  1 ====== starting new request req=0x7fc1d8e4bed0 =====
2024-11-19 09:57:26.896 7fc1d8e54700  1 ====== req done req=0x7fc1d8e4bed0 op status=0 http_status=404 latency=0.001s ======
2024-11-19 09:57:26.896 7fc1d8e54700  1 civetweb: 0x55c0936269d8: 192.168.0.103 - - [19/Nov/2024:09:57:26 +0800] "POST /mooncake_postgres_pg_mooncake_test_16419/6834f954-a236-4605-ad73-d6cece3c19c4.parquet?uploads= HTTP/1.1" 404 432 - cpp-httplib/0.14.3
2024-11-19 09:57:41.692 7fc1dd65d700  0 could not get bucket info for bucket=test[8e5c34ee-5bcc-46d6-bb56-9e935a943ce8.14140.1] r=-2
2024-11-19 09:57:41.692 7fc1dd65d700  0 WARNING: sync_bucket() returned r=-2

the bucket was like (the content of the json file is log of CREATE TABLE),

# s3cmd ls s3://test
                          DIR  s3://test/mooncake_postgres_pg_mooncake_test_16419/
# s3cmd ls s3://test/mooncake_postgres_pg_mooncake_test_16419/
                          DIR  s3://test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/
# s3cmd ls s3://test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/
2024-11-19 01:09         1728  s3://test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/00000000000000000000.json

OS:

RHEL 8

pg_mooncake Version:

f1818d436328fef51c5fa418eb3b43d21ac128cd

Postgres Version:

16.4

Are you using pg_mooncake Docker, Neon, or the extension standalone?

pg_mooncake extension standalone

zhousun commented 2 days ago

yea. ceph radosgw by default is using Path-Style Addressing. while we are working on some general fix, can you try:

update mooncake.secrets set duckdb_query='CREATE SECRET "duckdb_secret_test" (TYPE S3, KEY_ID ''J99FFEUCNV428W9DXS5M'', SECRET ''0nGNMIACzFOpRgD1XuqbmtOhv7V3mQWIPT7
1eAOt'', ENDPOINT ''192.168.2.201'', USE_SSL FALSE, URL_STYLE ''PATH'')';

and see if it works?

bpint commented 2 days ago

thanks for your help. URL_STYLE worked to some extend. pg_mooncake did not request domain test.192.168.2.201 any more. however, there was still no bucket part in the requested url.

# psql -U postgres -h 127.0.0.1
psql (16.4)
Type "help" for help.

postgres=# CREATE EXTENSION pg_mooncake;
CREATE EXTENSION
postgres=# SELECT mooncake.create_secret('test', 'S3', 'J99FFEUCNV428W9DXS5M', '0nGNMIACzFOpRgD1XuqbmtOhv7V3mQWIPT71eAOt', '{"ENDPOINT":"192.168.2.201", "USE_SSL": 0}');
 create_secret 
---------------

(1 row)

postgres=# update mooncake.secrets set duckdb_query='CREATE SECRET "duckdb_secret_test" (TYPE S3, KEY_ID ''J99FFEUCNV428W9DXS5M'', SECRET ''0nGNMIACzFOpRgD1XuqbmtOhv7V3mQWIPT71eAOt'', ENDPOINT ''192.168.2.201'', USE_SSL FALSE, URL_STYLE ''PATH'')';
UPDATE 1
postgres=# SELECT duckdb_query FROM mooncake.secrets;
                                                                                       duckdb_query                                            

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 CREATE SECRET "duckdb_secret_test" (TYPE S3, KEY_ID 'J99FFEUCNV428W9DXS5M', SECRET '0nGNMIACzFOpRgD1XuqbmtOhv7V3mQWIPT71eAOt', ENDPOINT '192.168.2.201', USE_SSL FALSE, URL_STYLE 'PATH')
(1 row)

postgres=# SET mooncake.default_bucket = 's3://test';
SET
postgres=# CREATE TABLE pg_mooncake_test(id BIGINT, log TEXT, ts TIMESTAMP) USING columnstore;
CREATE TABLE
postgres=# INSERT INTO pg_mooncake_test VALUES (1, 'test', '2024-01-01 08:00:00');
ERROR:  (PGDuckDB/ExecuteQuery) Invalid Error: Unexpected response while initializing S3 multipart upload
postgres=#

ceph radosgw log,

2024-11-20 09:07:38.854 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 09:07:38.858 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=404 latency=0.004s ======
2024-11-20 09:07:38.858 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:09:07:38 +0800] "HEAD /mooncake_postgres_pg_mooncake_test_16423/adb75c4a-3f2a-4060-bb86-616e1f972c5a.parquet HTTP/1.1" 404 204 - cpp-httplib/0.14.3
2024-11-20 09:07:39.020 7fc1d8e54700  1 ====== starting new request req=0x7fc1d8e4bed0 =====
2024-11-20 09:07:39.020 7fc1d8e54700  1 ====== req done req=0x7fc1d8e4bed0 op status=0 http_status=404 latency=0s ======
2024-11-20 09:07:39.020 7fc1d8e54700  1 civetweb: 0x55c0936269d8: 192.168.0.103 - - [20/Nov/2024:09:07:38 +0800] "POST /mooncake_postgres_pg_mooncake_test_16423/adb75c4a-3f2a-4060-bb86-616e1f972c5a.parquet?uploads= HTTP/1.1" 404 454 - cpp-httplib/0.14.3
zhousun commented 2 days ago

urrr. seems we are hitting some duckdb issue. try this:

update mooncake.secrets set duckdb_query='CREATE SECRET "duckdb_secret_test" (TYPE S3, KEY_ID ''J99FFEUCNV428W9DXS5M'', SECRET ''0nGNMIACzFOpRgD1XuqbmtOhv7V3mQWIPT7
1eAOt'', ENDPOINT ''192.168.2.201/test'', USE_SSL FALSE, URL_STYLE ''PATH'')';
bpint commented 2 days ago

great! it worked.

though CREATE TABLE and INSERT were slow on s3 storage, say 10 seconds, they completed with no errors.

# psql -U postgres -h 127.0.0.1
psql (16.4)
Type "help" for help.

postgres=# \timing
Timing is on.
postgres=# CREATE EXTENSION pg_mooncake;
CREATE EXTENSION
Time: 48.550 ms
postgres=# SELECT mooncake.create_secret('test', 'S3', 'J99FFEUCNV428W9DXS5M', '0nGNMIACzFOpRgD1XuqbmtOhv7V3mQWIPT71eAOt', '{"ENDPOINT":"192.168.2.201", "USE_SSL": 0}');
 create_secret 
---------------

(1 row)

Time: 7.540 ms
postgres=# update mooncake.secrets set duckdb_query='CREATE SECRET "duckdb_secret_test" (TYPE S3, KEY_ID ''J99FFEUCNV428W9DXS5M'', SECRET ''0nGNMIACzFOpRgD1XuqbmtOhv7V3mQWIPT71eAOt'', ENDPOINT ''192.168.2.201/test'', USE_SSL FALSE, URL_STYLE ''PATH'')';
UPDATE 1
Time: 6.783 ms
postgres=# SET mooncake.default_bucket = 's3://test';
SET
Time: 0.337 ms
postgres=# CREATE TABLE pg_mooncake_test(id BIGINT, log TEXT, ts TIMESTAMP) USING columnstore;
CREATE TABLE
Time: 9731.790 ms (00:09.732)
postgres=# INSERT INTO pg_mooncake_test VALUES (1, 'test', '2024-01-01 08:00:00');
INSERT 0 1
Time: 11142.336 ms (00:11.142)
postgres=# SELECT * FROM pg_mooncake_test;
 id | log  |         ts          
----+------+---------------------
  1 | test | 2024-01-01 08:00:00
(1 row)

Time: 3.287 ms
postgres=#

s3 bucket was like,

# s3cmd ls s3://test/mooncake_postgres_pg_mooncake_test_16419/
                          DIR  s3://test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/
2024-11-20 02:15          435  s3://test/mooncake_postgres_pg_mooncake_test_16419/d6eef5aa-8c9c-4285-b532-836865a9e934.parquet

local cache,

# ls -l /var/lib/pgsql/data/mooncake_local_cache/
total 4
-rw------- 1 postgres postgres 435 Nov 20 10:15 d6eef5aa-8c9c-4285-b532-836865a9e934.parquet

ceph side,

2024-11-20 10:15:08.310 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:08.323 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.012s ======
2024-11-20 10:15:08.323 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:08 +0800] "GET /test?list-type=2&prefix=mooncake_postgres_pg_mooncake_test_16419%2F_delta_log%2F HTTP/1.1" 200 480 - object_store/0.10.2
2024-11-20 10:15:08.387 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:08.587 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.2s ======
2024-11-20 10:15:08.587 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:08 +0800] "PUT /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/_commit_eb0708e9-ef72-4e59-aa6f-0038710252f3.json.tmp HTTP/1.1" 200 240 - object_store/0.10.2
2024-11-20 10:15:08.763 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:08.931 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.168s ======
2024-11-20 10:15:08.931 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:08 +0800] "PUT /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/00000000000000000000.json HTTP/1.1" 200 420 - object_store/0.10.2
2024-11-20 10:15:09.016 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:09.171 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=204 latency=0.155s ======
2024-11-20 10:15:09.171 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:08 +0800] "DELETE /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/_commit_eb0708e9-ef72-4e59-aa6f-0038710252f3.json.tmp HTTP/1.1" 204 131 - object_store/0.10.2
2024-11-20 10:15:09.558 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:09.559 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=404 latency=0.001s ======
2024-11-20 10:15:09.559 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:08 +0800] "GET /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/_last_checkpoint HTTP/1.1" 404 415 - object_store/0.10.2
2024-11-20 10:15:09.716 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:09.720 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.004s ======
2024-11-20 10:15:09.720 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:08 +0800] "GET /test?list-type=2&prefix=mooncake_postgres_pg_mooncake_test_16419%2F_delta_log%2F&start-after=mooncake_postgres_pg_mooncake_test_16419%2F_delta_log%2F00000000000000000000 HTTP/1.1" 200 920 - object_store/0.10.2
2024-11-20 10:15:10.013 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:10.018 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.005s ======
2024-11-20 10:15:10.019 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:08 +0800] "GET /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/00000000000000000000.json HTTP/1.1" 200 2099 - object_store/0.10.2
2024-11-20 10:15:10.091 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:10.094 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.003s ======
2024-11-20 10:15:10.094 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:08 +0800] "GET /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/00000000000000000000.json HTTP/1.1" 200 2099 - object_store/0.10.2
2024-11-20 10:15:10.124 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:10.125 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=404 latency=0.001s ======
2024-11-20 10:15:10.126 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:08 +0800] "GET /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/_last_checkpoint HTTP/1.1" 404 415 - object_store/0.10.2
2024-11-20 10:15:10.237 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:10.241 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.004s ======
2024-11-20 10:15:10.241 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:08 +0800] "GET /test?list-type=2&prefix=mooncake_postgres_pg_mooncake_test_16419%2F_delta_log%2F&start-after=mooncake_postgres_pg_mooncake_test_16419%2F_delta_log%2F00000000000000000000 HTTP/1.1" 200 920 - object_store/0.10.2
2024-11-20 10:15:10.350 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:10.353 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.003s ======
2024-11-20 10:15:10.353 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:08 +0800] "GET /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/00000000000000000000.json HTTP/1.1" 200 2099 - object_store/0.10.2
2024-11-20 10:15:10.385 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:10.388 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.003s ======
2024-11-20 10:15:10.388 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:08 +0800] "GET /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/00000000000000000000.json HTTP/1.1" 200 2099 - object_store/0.10.2
2024-11-20 10:15:20.352 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:20.355 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=404 latency=0.003s ======
2024-11-20 10:15:20.355 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:20 +0800] "HEAD /test/mooncake_postgres_pg_mooncake_test_16419/d6eef5aa-8c9c-4285-b532-836865a9e934.parquet HTTP/1.1" 404 204 - cpp-httplib/0.14.3
2024-11-20 10:15:20.517 7fc1d8e54700  1 ====== starting new request req=0x7fc1d8e4bed0 =====
2024-11-20 10:15:20.661 7fc1d8e54700  1 ====== req done req=0x7fc1d8e4bed0 op status=0 http_status=200 latency=0.144s ======
2024-11-20 10:15:20.661 7fc1d8e54700  1 civetweb: 0x55c0936269d8: 192.168.0.103 - - [20/Nov/2024:10:15:20 +0800] "POST /test/mooncake_postgres_pg_mooncake_test_16419/d6eef5aa-8c9c-4285-b532-836865a9e934.parquet?uploads= HTTP/1.1" 200 495 - cpp-httplib/0.14.3
2024-11-20 10:15:20.830 7fc1d8e54700  1 ====== starting new request req=0x7fc1d8e4bed0 =====
2024-11-20 10:15:21.705 7fc1d8e54700  1 ====== req done req=0x7fc1d8e4bed0 op status=0 http_status=200 latency=0.875s ======
2024-11-20 10:15:21.705 7fc1d8e54700  1 civetweb: 0x55c0936269d8: 192.168.0.103 - - [20/Nov/2024:10:15:20 +0800] "PUT /test/mooncake_postgres_pg_mooncake_test_16419/d6eef5aa-8c9c-4285-b532-836865a9e934.parquet?partNumber=1&uploadId=2~SkcXkRkXLylOlCHLOPk2MQAyUEpNAqI HTTP/1.1" 200 240 - cpp-httplib/0.14.3
2024-11-20 10:15:21.792 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:22.229 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.437s ======
2024-11-20 10:15:22.229 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:21 +0800] "POST /test/mooncake_postgres_pg_mooncake_test_16419/d6eef5aa-8c9c-4285-b532-836865a9e934.parquet?uploadId=2~SkcXkRkXLylOlCHLOPk2MQAyUEpNAqI HTTP/1.1" 200 605 - cpp-httplib/0.14.3
2024-11-20 10:15:29.548 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:29.551 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=404 latency=0.003s ======
2024-11-20 10:15:29.551 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:29 +0800] "GET /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/_last_checkpoint HTTP/1.1" 404 415 - object_store/0.10.2
2024-11-20 10:15:29.675 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:29.681 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.006s ======
2024-11-20 10:15:29.681 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:29 +0800] "GET /test?list-type=2&prefix=mooncake_postgres_pg_mooncake_test_16419%2F_delta_log%2F&start-after=mooncake_postgres_pg_mooncake_test_16419%2F_delta_log%2F00000000000000000000 HTTP/1.1" 200 920 - object_store/0.10.2
2024-11-20 10:15:29.797 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:29.800 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.003s ======
2024-11-20 10:15:29.800 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:29 +0800] "GET /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/00000000000000000000.json HTTP/1.1" 200 2099 - object_store/0.10.2
2024-11-20 10:15:29.915 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:29.917 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.002s ======
2024-11-20 10:15:29.917 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:29 +0800] "GET /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/00000000000000000000.json HTTP/1.1" 200 2099 - object_store/0.10.2
2024-11-20 10:15:30.031 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:30.186 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.155s ======
2024-11-20 10:15:30.186 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:29 +0800] "PUT /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/_commit_ed9eea80-2e13-4160-8c04-2dd233c2b7d4.json.tmp HTTP/1.1" 200 240 - object_store/0.10.2
2024-11-20 10:15:30.196 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:30.396 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.2s ======
2024-11-20 10:15:30.396 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:29 +0800] "PUT /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/00000000000000000001.json HTTP/1.1" 200 420 - object_store/0.10.2
2024-11-20 10:15:30.520 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:30.666 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=204 latency=0.146s ======
2024-11-20 10:15:30.667 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:30 +0800] "DELETE /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/_commit_ed9eea80-2e13-4160-8c04-2dd233c2b7d4.json.tmp HTTP/1.1" 204 131 - object_store/0.10.2
2024-11-20 10:15:30.724 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:30.726 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=404 latency=0.002s ======
2024-11-20 10:15:30.726 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:30 +0800] "GET /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/_last_checkpoint HTTP/1.1" 404 415 - object_store/0.10.2
2024-11-20 10:15:30.879 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:30.884 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.004s ======
2024-11-20 10:15:30.884 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:30 +0800] "GET /test?list-type=2&prefix=mooncake_postgres_pg_mooncake_test_16419%2F_delta_log%2F&start-after=mooncake_postgres_pg_mooncake_test_16419%2F_delta_log%2F00000000000000000001 HTTP/1.1" 200 919 - object_store/0.10.2
2024-11-20 10:15:30.992 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:30.995 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.003s ======
2024-11-20 10:15:30.995 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:30 +0800] "GET /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/00000000000000000001.json HTTP/1.1" 200 762 - object_store/0.10.2
2024-11-20 10:15:31.109 7fc1d9655700  1 ====== starting new request req=0x7fc1d964ced0 =====
2024-11-20 10:15:31.114 7fc1d9655700  1 ====== req done req=0x7fc1d964ced0 op status=0 http_status=200 latency=0.005s ======
2024-11-20 10:15:31.114 7fc1d9655700  1 civetweb: 0x55c093626000: 192.168.0.103 - - [20/Nov/2024:10:15:30 +0800] "GET /test/mooncake_postgres_pg_mooncake_test_16419/_delta_log/00000000000000000001.json HTTP/1.1" 200 762 - object_store/0.10.2