Docker build is slow

[yushangdi]

I encountered the same problem as below, ./scripts/dockerbuild.sh is very slow.

2023年12月12日 13:49，zqxjjj @.***> 写道：

Looks like all messages are warnings. Let's figure out how to resolve these warnings.

— Reply to this email directly, view it on GitHub https://github.com/microsoft/MSVBASE/issues/9#issuecomment-1851343998, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOKP45XI4H5GDBJZ6JOMP73YI7WANAVCNFSM6AAAAABAHLBC6GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJRGM2DGOJZHA. You are receiving this because you authored the thread.

Ok, there is an another problem, the docker build is too slow. Can you provide a local way to build this repo?

Originally posted by @JackTan25 in https://github.com/microsoft/MSVBASE/issues/9#issuecomment-1851349181

[zqxjjj]

Thanks for reporting. After evaluation, we found that it takes less than 10 minutes to build from scratch. We think it is reasonable since it needs to compile boost and postgresql.

I am not sure why it takes too long time for you. If you could offer more details, we can figure out the issue better.

[yushangdi]

Hi, thanks for your reply. I tried again on another machine and it builds in a reasonable time (similar to yours) now. I'm not sure what's wrong with the first machine I tried, which took over 1hr and made no progress. It's a 12-core machine with ubuntu installed.

[yushangdi]

oh actually I figured out the why it didn't work the first time. I didn't do sudo.

If sudo is not used, it will not report any error. It will just hang there.

[yushangdi]

Also I needed to copy the build command and run it directly in terminal, otherwise I got an error due to group id 0 is already used (looks like group id is not passed in successfully).

[JackTan25]

hi, @yushangdi I succeed on the repo docker build,And I can use hnsw normally, but the spann index is bad, I get below:

postgres=# create table t4(a float[1]);
CREATE TABLE
postgres=# alter table t4 alter a set storage plain;
ALTER TABLE
postgres=# insert into t4 values('{1}');
INSERT 0 1
postgres=# insert into t4 values('{2}');
INSERT 0 1
postgres=# insert into t4 values('{3}');
INSERT 0 1
postgres=# insert into t4 values('{4}');
INSERT 0 1
postgres=# create index a_spann_index2 on t4 using spann(a spann_vector_l2_ops);
INFO:  Saved to /u02/pgdata/13/base/12591/
CREATE INDEX
postgres=# set enable_seqscan = off;
SET
postgres=# explain select * from t4 order by a <-> '{1}' limit 10;
                                      QUERY PLAN                                      
--------------------------------------------------------------------------------------
 Limit  (cost=8.03..8.04 rows=1 width=40)
   ->  Sort  (cost=8.03..8.04 rows=1 width=40)
         Sort Key: ((a <-> '{1}'::double precision[]))
         ->  Index Scan using a_spann_index2 on t4  (cost=0.00..8.02 rows=1 width=40)
               Order By: (a <-> '{1}'::double precision[])
(5 rows)

postgres=# select * from t4 order by a <-> '{1}' limit 10;
server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
The connection to the server was lost. Attempting reset: Failed.
!?> 

The postgres process crash, that's very strange, But in Vbase paper, it acctually can run and get good performance.Maybe the author upload the source code by mistake. @zqxjjj @yushangdi

[JackTan25]

The log is below:

2024-02-27 10:22:39.430 UTC [45] LOG: server process (PID 101) was terminated by signal 11: Segmentation fault
2024-02-27 10:22:39.430 UTC [45] DETAIL: Failed process was running: select * from t1 order by a <-> '{1}' limit 10;
2024-02-27 10:22:39.430 UTC [45] LOG: terminating any other active server processes
2024-02-27 10:22:39.430 UTC [72] WARNING: terminating connection because of crash of another server process
2024-02-27 10:22:39.430 UTC [72] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2024-02-27 10:22:39.430 UTC [72] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2024-02-27 10:22:39.432 UTC [103] FATAL: the database system is in recovery mode
2024-02-27 10:22:39.434 UTC [45] LOG: all server processes terminated; reinitializing
2024-02-27 10:22:39.478 UTC [104] LOG: database system was interrupted; last known up at 2024-02-27 10:12:27 UTC
2024-02-27 10:22:39.551 UTC [104] LOG: database system was not properly shut down; automatic recovery in progress
2024-02-27 10:22:39.553 UTC [104] LOG: redo starts at 0/40E00168
2024-02-27 10:22:39.553 UTC [104] LOG: invalid record length at 0/40E001A0: wanted 24, got 0
2024-02-27 10:22:39.553 UTC [104] LOG: redo done at 0/40E00168
2024-02-27 10:22:39.561 UTC [45] LOG: database system is ready to accept connections