search

wandb / server

W&B Server is the self hosted version of Weights & Biases
MIT License
263 stars 21 forks source link

wandb server start --upgarde hard to migrate database #94

Open Jeffrey28 opened 2 years ago

Jeffrey28 commented 2 years ago 
docker logs -f wandb-local
*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/01_enable-services.sh...
*** Copying services to runit
*** Copying jobber template
*** Enabling production mode
*** Running /etc/my_init.d/02_load-settings.sh...
*** Loading settings...
*** Booting runit daemon...
*** Runit started as PID 84
*** Starting wandb servers...
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/0519aff9-4e68-40de-a96d-78fcd159aea0: permission denied
./run: line 1: kill: (241) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/0519aff9-4e68-40de-a96d-78fcd159aea0: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/b04bab63-aa51-4d95-b3e5-6671cae44043: permission denied
./run: line 1: kill: (433) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/0519aff9-4e68-40de-a96d-78fcd159aea0: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/b04bab63-aa51-4d95-b3e5-6671cae44043: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/f1527e05-8192-4605-a7f2-d1bcecceed41: permission denied
./run: line 1: kill: (607) - No such process
*** Migrating database...
panic: dial tcp 127.0.0.1:3306: connect: connection refused
Jeffrey28 commented 2 years ago

and another problem: After I tried cat /vol/env/users.htpasswd in the docker exec -it wandb-local bash wandb_local is not the right password?

Jeffrey28 commented 2 years ago

image

vanpelt commented 2 years ago

Hey @Jeffrey28 we just released a new version of the server that should address this problem. Can you try upgrading again?

Jeffrey28 commented 2 years ago

Actually no. The latest two versions both have the same issue.

vanpelt commented 2 years ago

@Jeffrey28 I've verified 0.22.0 works fine when upgrading from 0.20.0. There was a regression in 0.21.0. Can you try running:

docker pull wandb/local:latest
docker stop wandb-local
wandb server start

If that doesn't work, can you share the output of docker logs -f wandb-local?

Jeffrey28 commented 2 years ago

I‘m in 0.21.0. It works only after I delete the volume. But if I try upgrading to 0.22.0, it failed.

docker logs -f wandb-local
*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/01_enable-services.sh...
*** Copying services to runit
*** Copying jobber template
*** Enabling production mode
*** Running /etc/my_init.d/02_load-settings.sh...
*** Loading settings...
*** Booting runit daemon...
*** Runit started as PID 40
*** Starting wandb servers...
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/ab31d888-89d6-4260-9dfc-c6a9a779d474: permission denied
./run: line 1: kill: (118) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/ab31d888-89d6-4260-9dfc-c6a9a779d474: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/1a5e6dae-51fe-459b-bfff-a0674090640e: permission denied
./run: line 1: kill: (279) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/ab31d888-89d6-4260-9dfc-c6a9a779d474: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/1a5e6dae-51fe-459b-bfff-a0674090640e: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/0c8a61a9-9a9f-43ec-89b9-6643dadf6ef4: permission denied
./run: line 1: kill: (392) - No such process
*** Minio still starting up...
*** Migrating database...
panic: dial tcp 127.0.0.1:3306: connect: connection refused

goroutine 1 [running]:
github.com/wandb/core/services/gorilla/cmd.(*migrateCommander).MainCmd(0xc000d143e0, {0xc000c88540, 0x2, 0x2})
    /mnt/ramdisk/core/services/gorilla/cmd/migrate.go:88 +0x9b4
main.main()
    /mnt/ramdisk/core/services/gorilla/cmd/megabinary/main.go:57 +0x227
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/ab31d888-89d6-4260-9dfc-c6a9a779d474: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/1a5e6dae-51fe-459b-bfff-a0674090640e: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/0c8a61a9-9a9f-43ec-89b9-6643dadf6ef4: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d87586eb-d242-4200-83c7-e222b003e207: permission denied
./run: line 1: kill: (510) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/1a5e6dae-51fe-459b-bfff-a0674090640e: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/0c8a61a9-9a9f-43ec-89b9-6643dadf6ef4: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d87586eb-d242-4200-83c7-e222b003e207: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/4f95c63e-6211-4dd4-b57f-18c2bf5fe5ba: permission denied
./run: line 1: kill: (706) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/0c8a61a9-9a9f-43ec-89b9-6643dadf6ef4: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d87586eb-d242-4200-83c7-e222b003e207: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/4f95c63e-6211-4dd4-b57f-18c2bf5fe5ba: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/56af9bcf-d2ee-4e75-a22e-e6c200234016: permission denied
./run: line 1: kill: (822) - No such process
*** Minio still starting up...
*** Still waiting...
*** Migrating database...
panic: dial tcp 127.0.0.1:3306: connect: connection refused

goroutine 1 [running]:
github.com/wandb/core/services/gorilla/cmd.(*migrateCommander).MainCmd(0xc0001243b0, {0xc00012a2a0, 0x2, 0x2})
    /mnt/ramdisk/core/services/gorilla/cmd/migrate.go:88 +0x9b4
main.main()
    /mnt/ramdisk/core/services/gorilla/cmd/megabinary/main.go:57 +0x227
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d87586eb-d242-4200-83c7-e222b003e207: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/4f95c63e-6211-4dd4-b57f-18c2bf5fe5ba: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/56af9bcf-d2ee-4e75-a22e-e6c200234016: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/f5cb6796-a0e1-453e-a3b0-23b44e3556b3: permission denied
./run: line 1: kill: (939) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/4f95c63e-6211-4dd4-b57f-18c2bf5fe5ba: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/56af9bcf-d2ee-4e75-a22e-e6c200234016: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/f5cb6796-a0e1-453e-a3b0-23b44e3556b3: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/110ebc38-c955-4775-a5a0-e0060247909d: permission denied
./run: line 1: kill: (1091) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/56af9bcf-d2ee-4e75-a22e-e6c200234016: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/f5cb6796-a0e1-453e-a3b0-23b44e3556b3: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/110ebc38-c955-4775-a5a0-e0060247909d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/089850a3-5e56-4e71-96ba-13dfa330fd63: permission denied
./run: line 1: kill: (1207) - No such process
*** Minio still starting up...
*** Still waiting...
*** Migrating database...
panic: dial tcp 127.0.0.1:3306: connect: connection refused

goroutine 1 [running]:
github.com/wandb/core/services/gorilla/cmd.(*migrateCommander).MainCmd(0xc00013c370, {0xc0000e2380, 0x2, 0x2})
    /mnt/ramdisk/core/services/gorilla/cmd/migrate.go:88 +0x9b4
main.main()
    /mnt/ramdisk/core/services/gorilla/cmd/megabinary/main.go:57 +0x227
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/f5cb6796-a0e1-453e-a3b0-23b44e3556b3: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/110ebc38-c955-4775-a5a0-e0060247909d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/089850a3-5e56-4e71-96ba-13dfa330fd63: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d11c1bd9-0d43-44f5-8378-c01f708cf70d: permission denied
./run: line 1: kill: (1327) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/110ebc38-c955-4775-a5a0-e0060247909d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/089850a3-5e56-4e71-96ba-13dfa330fd63: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d11c1bd9-0d43-44f5-8378-c01f708cf70d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/abf0e4a9-ec36-4d48-8ee1-13e9e5db74f0: permission denied
./run: line 1: kill: (1468) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/089850a3-5e56-4e71-96ba-13dfa330fd63: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d11c1bd9-0d43-44f5-8378-c01f708cf70d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/abf0e4a9-ec36-4d48-8ee1-13e9e5db74f0: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/5b2e2203-f758-4a9a-a686-ba8806e0fe9c: permission denied
./run: line 1: kill: (1628) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d11c1bd9-0d43-44f5-8378-c01f708cf70d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/abf0e4a9-ec36-4d48-8ee1-13e9e5db74f0: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/5b2e2203-f758-4a9a-a686-ba8806e0fe9c: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d452a52b-aac4-4b76-b42a-b64a84405e85: permission denied
./run: line 1: kill: (1742) - No such process
*** Minio still starting up...
Jeffrey28 commented 2 years ago

Recent versions cause me to fear upgrading

vanpelt commented 2 years ago

We understand your fear and are adding more tests to prevent upgrade failures like this in the future.

0.21.0 had a regression which caused volumes created previously to fail upgrading. If your volume was created in any version prior to 0.21.0, then upgrading to 0.22.0 would work. If you created a fresh volume with 0.21.0 you'll need to either delete the volume, or run the following after upgrading:

docker exec -it wandb-local sudo fix_permissions.sh /vol
docker restart wandb-local
exalate-issue-sync[bot] commented 2 years ago

WandB Internal User commented: Jeffrey28 commented: Actually no. The latest two versions both have the same issue.

exalate-issue-sync[bot] commented 2 years ago

WandB Internal User commented: Jeffrey28 commented: I‘m in 0.21.0. It works only after I delete the volume. But if I try upgrading, it failed.

docker logs -f wandb-local
*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/01_enable-services.sh...
*** Copying services to runit
*** Copying jobber template
*** Enabling production mode
*** Running /etc/my_init.d/02_load-settings.sh...
*** Loading settings...
*** Booting runit daemon...
*** Runit started as PID 40
*** Starting wandb servers...
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/ab31d888-89d6-4260-9dfc-c6a9a779d474: permission denied
./run: line 1: kill: (118) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/ab31d888-89d6-4260-9dfc-c6a9a779d474: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/1a5e6dae-51fe-459b-bfff-a0674090640e: permission denied
./run: line 1: kill: (279) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/ab31d888-89d6-4260-9dfc-c6a9a779d474: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/1a5e6dae-51fe-459b-bfff-a0674090640e: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/0c8a61a9-9a9f-43ec-89b9-6643dadf6ef4: permission denied
./run: line 1: kill: (392) - No such process
*** Minio still starting up...
*** Migrating database...
panic: dial tcp 127.0.0.1:3306: connect: connection refused

goroutine 1 [running]:
github.com/wandb/core/services/gorilla/cmd.(*migrateCommander).MainCmd(0xc000d143e0, {0xc000c88540, 0x2, 0x2})
    /mnt/ramdisk/core/services/gorilla/cmd/migrate.go:88 +0x9b4
main.main()
    /mnt/ramdisk/core/services/gorilla/cmd/megabinary/main.go:57 +0x227
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/ab31d888-89d6-4260-9dfc-c6a9a779d474: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/1a5e6dae-51fe-459b-bfff-a0674090640e: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/0c8a61a9-9a9f-43ec-89b9-6643dadf6ef4: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d87586eb-d242-4200-83c7-e222b003e207: permission denied
./run: line 1: kill: (510) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/1a5e6dae-51fe-459b-bfff-a0674090640e: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/0c8a61a9-9a9f-43ec-89b9-6643dadf6ef4: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d87586eb-d242-4200-83c7-e222b003e207: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/4f95c63e-6211-4dd4-b57f-18c2bf5fe5ba: permission denied
./run: line 1: kill: (706) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/0c8a61a9-9a9f-43ec-89b9-6643dadf6ef4: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d87586eb-d242-4200-83c7-e222b003e207: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/4f95c63e-6211-4dd4-b57f-18c2bf5fe5ba: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/56af9bcf-d2ee-4e75-a22e-e6c200234016: permission denied
./run: line 1: kill: (822) - No such process
*** Minio still starting up...
*** Still waiting...
*** Migrating database...
panic: dial tcp 127.0.0.1:3306: connect: connection refused

goroutine 1 [running]:
github.com/wandb/core/services/gorilla/cmd.(*migrateCommander).MainCmd(0xc0001243b0, {0xc00012a2a0, 0x2, 0x2})
    /mnt/ramdisk/core/services/gorilla/cmd/migrate.go:88 +0x9b4
main.main()
    /mnt/ramdisk/core/services/gorilla/cmd/megabinary/main.go:57 +0x227
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d87586eb-d242-4200-83c7-e222b003e207: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/4f95c63e-6211-4dd4-b57f-18c2bf5fe5ba: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/56af9bcf-d2ee-4e75-a22e-e6c200234016: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/f5cb6796-a0e1-453e-a3b0-23b44e3556b3: permission denied
./run: line 1: kill: (939) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/4f95c63e-6211-4dd4-b57f-18c2bf5fe5ba: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/56af9bcf-d2ee-4e75-a22e-e6c200234016: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/f5cb6796-a0e1-453e-a3b0-23b44e3556b3: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/110ebc38-c955-4775-a5a0-e0060247909d: permission denied
./run: line 1: kill: (1091) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/56af9bcf-d2ee-4e75-a22e-e6c200234016: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/f5cb6796-a0e1-453e-a3b0-23b44e3556b3: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/110ebc38-c955-4775-a5a0-e0060247909d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/089850a3-5e56-4e71-96ba-13dfa330fd63: permission denied
./run: line 1: kill: (1207) - No such process
*** Minio still starting up...
*** Still waiting...
*** Migrating database...
panic: dial tcp 127.0.0.1:3306: connect: connection refused

goroutine 1 [running]:
github.com/wandb/core/services/gorilla/cmd.(*migrateCommander).MainCmd(0xc00013c370, {0xc0000e2380, 0x2, 0x2})
    /mnt/ramdisk/core/services/gorilla/cmd/migrate.go:88 +0x9b4
main.main()
    /mnt/ramdisk/core/services/gorilla/cmd/megabinary/main.go:57 +0x227
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/f5cb6796-a0e1-453e-a3b0-23b44e3556b3: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/110ebc38-c955-4775-a5a0-e0060247909d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/089850a3-5e56-4e71-96ba-13dfa330fd63: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d11c1bd9-0d43-44f5-8378-c01f708cf70d: permission denied
./run: line 1: kill: (1327) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/110ebc38-c955-4775-a5a0-e0060247909d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/089850a3-5e56-4e71-96ba-13dfa330fd63: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d11c1bd9-0d43-44f5-8378-c01f708cf70d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/abf0e4a9-ec36-4d48-8ee1-13e9e5db74f0: permission denied
./run: line 1: kill: (1468) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/089850a3-5e56-4e71-96ba-13dfa330fd63: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d11c1bd9-0d43-44f5-8378-c01f708cf70d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/abf0e4a9-ec36-4d48-8ee1-13e9e5db74f0: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/5b2e2203-f758-4a9a-a686-ba8806e0fe9c: permission denied
./run: line 1: kill: (1628) - No such process
*** Minio still starting up...
*** Still waiting...
mc: <ERROR> Unable to get service status. Get "http://127.0.0.1:9000/minio/admin/v3/info": dial tcp 127.0.0.1:9000: connect: connection refused.
!!! Something's not right
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d11c1bd9-0d43-44f5-8378-c01f708cf70d: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/abf0e4a9-ec36-4d48-8ee1-13e9e5db74f0: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/5b2e2203-f758-4a9a-a686-ba8806e0fe9c: permission denied
WARNING: MINIO_ACCESS_KEY and MINIO_SECRET_KEY are deprecated.
         Please use MINIO_ROOT_USER and MINIO_ROOT_PASSWORD
ERROR Unable to initialize backend: mkdir /vol/minio/.minio.sys/tmp/d452a52b-aac4-4b76-b42a-b64a84405e85: permission denied
./run: line 1: kill: (1742) - No such process
*** Minio still starting up...
exalate-issue-sync[bot] commented 2 years ago

WandB Internal User commented: Jeffrey28 commented: Recent versions cause me to fear upgrading

exalate-issue-sync[bot] commented 2 years ago

WandB Internal User commented: vanpelt commented: We understand your fear and our adding more tests to prevent upgrade failures like this in the future.

0.21.0 had a regression which caused volumes created previously to fail upgrading. If your volume was created in any version prior to 0.21.0, then upgrading to 0.22.0 would work. If you created a fresh volume with 0.21.0 you'll need to either delete the volume, or run the following after upgrading:

docker exec -it wandb-local sudo fix_permissions.sh /vol
docker restart wandb-local
Jeffrey28 commented 2 years ago

Thanks, it works now with these two lines.

Jeffrey28 commented 2 years ago

But coming with the problem: image

Jeffrey28 commented 2 years ago

It has to remove the volume. But the data will be missing.

vanpelt commented 2 years ago

Sorry about the 2 lines not working, we'll investigate and reproduce the workspace error and provide lines that work. You should never lose data, that's unacceptable.

vanpelt commented 2 years ago

@Jeffrey28 I just verified lines that work when upgrading from a volume that was first mounted on version 0.21.0, to a server running 0.22.0: 

docker exec -it wandb-local sudo chown -R wandb /vol
wandb server stop
wandb server start
exalate-issue-sync[bot] commented 2 years ago

WandB Internal User commented: Jeffrey28 commented:

Thanks, it works now with these two lines.

exalate-issue-sync[bot] commented 2 years ago

WandB Internal User commented: vanpelt commented: Sorry about the 2 lines not working, we'll investigate and reproduce the workspace error and provide lines that work. You should never lose data, that's unacceptable.

exalate-issue-sync[bot] commented 2 years ago

Luis Bergua commented: Hi Jeffrey, I just wanted to ask you if these lines worked for you when upgrading the 0.21.0 volume to 0.22.0

docker exec -it wandb-local sudo chown -R wandb /volwandb server stopwandb server start

Thanks!

Jeffrey28 commented 2 years ago

Luis Bergua commented: Hi Jeffrey, I just wanted to ask you if these lines worked for you when upgrading the 0.21.0 volume to 0.22.0

docker exec -it wandb-local sudo chown -R wandb /volwandb server stopwandb server start

Thanks!

It works but the data has some problems. Therefore, I have to reset the database.

luisbergua commented 2 years ago

Thanks for your answer @Jeffrey28, would you mind explaining me what kind of problems has the data and so I can try to help you or have you reset the database and it has solved these problems?

Jeffrey28 commented 2 years ago

After reseting the database, it works well now.

luisbergua commented 2 years ago

Great to hear it @Jeffrey28! I'll then close this issue as it seems to be solved. Thanks!

exalate-issue-sync[bot] commented 2 years ago

WandB Internal User commented: Jeffrey28 commented: After reseting the database, it works well now.

exalate-issue-sync[bot] commented 2 years ago

WandB Internal User commented: luisbergua commented: Great to hear it @Jeffrey28! I'll then close this issue as it seems to be solved. Thanks!