search

ydb-platform / nbs

Network Block Store
Apache License 2.0
52 stars 21 forks source link

docs: unable to run example #1

Closed ernado closed 1 year ago

ernado commented 1 year ago

Trying to follow https://github.com/ydb-platform/nbs/tree/main/example.

Encountered following problems:

0. Binary names

Probably the diskagentd is the new name for blockstore-disk-agent.

1. Local binaries

Example expects all binaries being available locally. I've created symlinks as a fix, like that:

ln -s /home/ernado/nbswork/build/cloud/blockstore/apps/disk_agent/diskagentd ./diskagentd

2. can't open "nbs/nbs-log.txt" with mode RdOnly|Seq

3-start_nbs.sh prints following after start:

2023-07-16-18-02-42 :BLOCKSTORE_SERVER INFO: /home/ernado/nbswork/nbs/cloud/blockstore/libs/daemon/common/bootstrap.cpp:198: NBS server version: 1689516977.main
main bootstrap start: (TFileError) (No such file or directory) /home/ernado/nbswork/nbs/util/system/file.cpp:857: can't open "nbs/nbs-sys.txt" with mode RdOnly|Seq (0x00000028)
2023-07-16-18-02-42 :BLOCKSTORE_SERVER INFO: /home/ernado/nbswork/nbs/cloud/blockstore/libs/daemon/common/bootstrap.cpp:828: Stopped Scheduler
2023-07-16-18-02-42 :BLOCKSTORE_SERVER INFO: /home/ernado/nbswork/nbs/cloud/blockstore/libs/daemon/common/bootstrap.cpp:831: Stopped BackgroundThreadPoo

For some reason, files are opened as readonly, but probably meant to be created by nbsd:

$NBSD \
    --domain             Root \
    --node-broker        localhost:$GRPC_PORT \
    --ic-port            $IC_PORT \
    --mon-port           $MON_PORT \
    --server-port        $SERVER_PORT \
    --data-server-port   $DATA_SERVER_PORT \
    --secure-server-port $SECURE_SERVER_PORT \
    --discovery-file     nbs/nbs-discovery.txt \
    --domains-file       nbs/nbs-domains.txt \
    --ic-file            nbs/nbs-ic.txt \
    --log-file           nbs/nbs-log.txt \
    --sys-file           nbs/nbs-sys.txt \
    --server-file        nbs/nbs-server.txt \
    --storage-file       nbs/nbs-storage.txt \
    --naming-file        nbs/nbs-names.txt \
    --diag-file          nbs/nbs-diag.txt \
    --auth-file          nbs/nbs-auth.txt \
    --dr-proxy-file      nbs/nbs-dr-proxy.txt \
    --service            kikimr \
    --load-configs-from-cms \
    --profile-file       logs/profile-log.bin \

I've created all these files via touch as a workaround.

3. can't open "nbs/nbs-log.txt" with mode RdOnly|Seq

2023-07-16-18-04-34 :BLOCKSTORE_SERVER ERROR: /home/ernado/nbswork/nbs/cloud/blockstore/libs/storage/core/manually_preempted_volumes.cpp:156: Failed to load manually preempted volumes: Failed to read preempted volumes list with error: (TFileError) (No such file or directory) /home/ernado/nbswork/nbs/util/system/file.cpp:857: can't open "/var/log/nbs-server/nbs-preempted-volumes.json" with mode RdOnly (0x00000008)

The /var/log/nbs-server/nbs-preempted-volumes.json file is required and the local path for it is not provided in config.

I've also created it manually.

4. requirement systemConfig.HasScheduler() failed

VERIFY failed (2023-07-16T18:12:14.181403+0300): 
  /home/ernado/nbswork/nbs/ydb/core/driver_lib/run/kikimr_services_initializers.cpp:594
  InitializeServices(): requirement systemConfig.HasScheduler() failed

Can't proceed or find workaround :(

Full log:

2023-07-16-18-13-01 :BLOCKSTORE_SERVER INFO: /home/ernado/nbswork/nbs/cloud/blockstore/libs/daemon/common/bootstrap.cpp:198: NBS server version: 1689516977.main
2023-07-16-18-13-01 :BLOCKSTORE_SERVER INFO: /home/ernado/nbswork/nbs/cloud/blockstore/libs/daemon/ydb/bootstrap.cpp:262: Configs initialized
2023-07-16-18-13-01 :BLOCKSTORE_SERVER INFO: /home/ernado/nbswork/nbs/cloud/storage/core/libs/kikimr/node.cpp:97: Trying to register dynamic node at "localhost:9001"
2023-07-16-18-13-01 :BLOCKSTORE_SERVER INFO: /home/ernado/nbswork/nbs/cloud/storage/core/libs/kikimr/node.cpp:124: Registered dynamic node at "localhost:9001" with address "127.0.1.1"
2023-07-16-18-13-01 :BLOCKSTORE_SERVER INFO: /home/ernado/nbswork/nbs/cloud/blockstore/libs/daemon/ydb/bootstrap.cpp:273: CMS configs initialized
2023-07-16-18-13-01 :BLOCKSTORE_SERVER INFO: /home/ernado/nbswork/nbs/cloud/blockstore/libs/daemon/ydb/bootstrap.cpp:292: ClientPercentiles initialized
2023-07-16-18-13-01 :BLOCKSTORE_SERVER INFO: /home/ernado/nbswork/nbs/cloud/blockstore/libs/daemon/ydb/bootstrap.cpp:307: StatsAggregator initialized
2023-07-16-18-13-01 :BLOCKSTORE_SERVER INFO: /home/ernado/nbswork/nbs/cloud/blockstore/libs/daemon/ydb/bootstrap.cpp:333: StatsUploader initialized
2023-07-16-18-13-01 :BLOCKSTORE_SERVER INFO: /home/ernado/nbswork/nbs/cloud/blockstore/libs/daemon/ydb/bootstrap.cpp:389: DiscoveryService initialized
2023-07-16-18-13-01 :BLOCKSTORE_SERVER INFO: /home/ernado/nbswork/nbs/cloud/blockstore/libs/daemon/ydb/bootstrap.cpp:396: TraceSerializer initialized
2023-07-16-18-13-01 :BLOCKSTORE_SERVER INFO: /home/ernado/nbswork/nbs/cloud/blockstore/libs/daemon/ydb/bootstrap.cpp:442: Allocator initialized
2023-07-16-18-13-01 :BLOCKSTORE_SERVER INFO: /home/ernado/nbswork/nbs/cloud/blockstore/libs/daemon/ydb/bootstrap.cpp:446: ProfileLog initialized
2023-07-16-18-13-01 :BLOCKSTORE_SERVER INFO: /home/ernado/nbswork/nbs/cloud/blockstore/libs/daemon/ydb/bootstrap.cpp:464: DigestGenerator initialized
2023-07-16-18-13-01 :BLOCKSTORE_SERVER INFO: /home/ernado/nbswork/nbs/cloud/blockstore/libs/daemon/ydb/bootstrap.cpp:470: LogbrokerService initialized
2023-07-16-18-13-01 :BLOCKSTORE_SERVER INFO: /home/ernado/nbswork/nbs/cloud/blockstore/libs/daemon/ydb/bootstrap.cpp:476: NotifyService initialized
UDFsDir is not specified, no dynamic UDFs will be loaded. 
VERIFY failed (2023-07-16T18:13:01.120651+0300): 
  /home/ernado/nbswork/nbs/ydb/core/driver_lib/run/kikimr_services_initializers.cpp:594
  InitializeServices(): requirement systemConfig.HasScheduler() failed
??+0 (0x8A08D57)
??+0 (0x8A005CB)
??+0 (0x8EF78A4)
??+0 (0x8EEFEB4)
??+0 (0x91D60A4)
??+0 (0x91C6C84)
??+0 (0x8EE9017)
??+0 (0x8A1EBB1)
??+0 (0x8AD2820)
??+0 (0x89953E9)
??+0 (0x89952E2)
??+0 (0x7F34C8609D90)
__libc_start_main+128 (0x7F34C8609E40)
??+0 (0x7F40025)
./3-start_nbs.sh: line 33: 1595767 Aborted                 (core dumped) $NBSD --domain Root --node-broker localhost:$GRPC_PORT --ic-port $IC_PORT --mon-port $MON_PORT --server-port $SERVER_PORT --data-server-port $DATA_SERVER_PORT --secure-server-port $SECURE_SERVER_PORT --discovery-file nbs/nbs-discovery.txt --domains-file nbs/nbs-domains.txt --ic-file nbs/nbs-ic.txt --log-file nbs/nbs-log.txt --sys-file nbs/nbs-sys.txt --server-file nbs/nbs-server.txt --storage-file nbs/nbs-storage.txt --naming-file nbs/nbs-names.txt --diag-file nbs/nbs-diag.txt --auth-file nbs/nbs-auth.txt --dr-proxy-file nbs/nbs-dr-proxy.txt --service kikimr --load-configs-from-cms --profile-file logs/profile-log.bin $@
qkrorlqr commented 1 year ago

Thank you, I will take a look at it till the end of this week

qkrorlqr commented 1 year ago

Sorry for taking so long to respond. I think I fixed the problems that you had spotted.

nbs/nbs-log.txt - it's actually a configuration file for our logger - I committed ydbd configs but forgot to commit nbsd configs, in particular - this config. I've just fixed it - added nbsd configs.

I also added a couple lines to 0-setup.sh that create symlinks from the folder where you run 0-setup.sh to the folders that contain those configs.

Could you please try to run the example once again and tell me whether everything works now? I will try to respond faster next time.

ernado commented 1 year ago

No problem, take as much time as you need.

I will try soon and let you know if everything is ok.

Thank you!

ernado commented 1 year ago

Now I'm able to run the example, thanks.