search

wandb / server

W&B Server is the self hosted version of Weights & Biases
MIT License
263 stars 21 forks source link

Finally I got in(cannot login and met wired login/signup error) #64

Open ericxsun opened 2 years ago

ericxsun commented 2 years ago

Through https://github.com/wandb/local/issues/64#issuecomment-1106030481, I got in

Pre:

With the following actions:

# start a new docker volume

docker stop wandb-local
docker volume rm wandb

docker run --rm -d -v wandb:/vol -p 8080:8080 --name wandb-local wandb/local

0) then, trigger the localhost:8080/logout

image

and then I can see thelogin and add license :

image

1) goto: Add license -> finally get http://localhost:8080/system-admin/

image

the Login links to http://localhost:8080/system-admin/login, a invalid link address (403 Forbidden)

2) goto: Login in 0) -> finally get http://localhost:8080/signup

image

click continue, I get the error:

image

3) set user and password through wandb bash cmd

docker exec -it wandb-local bash

wandb@66de73cf1e02:~$ cat /vol/env/users.htpasswd

only one line: local@wandb.com:.....

## create a user:
wandb@66de73cf1e02:~$ /usr/local/bin/local password test@gmail.com
Enter password for test@gmail.com:
xxxxx
User updated

and then goto http://localhost:8080/logout, and Login, it automatically jumps to the signup screen, and I get the same as described in 2) above.

4) try to restart after creation of the test@gmail.com in 3) and try again

docker stop wandb-local
docker run --rm -d -v wandb:/vol -p 8080:8080 --name wandb-local wandb/local

and then goto http://localhost:8080/logout, and Login, it automatically jumps to the signup screen, and I get the same as described in 2) above.

5) goto wandb shell and set something:

# rm local@wandb.com:.....   in /vol/env/users.htpasswd, then only one line left:
test@gmail.com:....

# restart mysql and outside the wandb docker, restart docker wandb local and re-login wandb shell, dive into mysql:
mysql -u wandb_local -p

select * from wandb_local.users;

the username of test@gmail.com is empty, then update it:
update wandb_local.users set username='test' where email = 'test@gmail';

# restart mysql and outside the wandb docker, restart docker wandb local
/etc/initd/mysql restart

docker run -e LOCAL_RESTORE=true --rm -d -v wandb:/vol -p 8080:8080 --name wandb-local wandb/local

re-goto http://localhost:8080/logout, and login, failed into signup page(it is different from the page in 2)

image

and re signup a new user test2, a wired error ocurred. signup username is test2 , but the error is for test, see the image:

image

wandb_local

Attached users in mysql (the user local is deleted image

It's really boring, for so many tries, I still cannot login successfully.

ref:

ericxsun commented 2 years ago

One more try:

step 1: create and start a new volume

docker stop wandb-local
docker volume rm wandb

docker run -e LOCAL_RESTORE=true --rm -d -v wandb:/vol -p 8080:8080 --name wandb-local wandb/local

step 2: login wandb docker shell and remove line local@wandb.com... in /vol/env/users.htpasswd

docker exec -it wandb-local bash

vi /vol/env/users.htpasswd  # remove line: local@wandb.com... 
cat /vol/env/users.htpasswd

image

NB: I guess the param -e LOCAL_RESTORE=true led to the line restore@wandb.com:....

step 3: create new user test@gmail.com

/usr/local/bin/local password test@gmail.com

cat /vol/env/users.htpasswd

image

and login mysql through mysql -u wandb_local -p to see some infos: image

step 4: restart mysql and docker wandb, 

# in wandb docker shell
/etc/init.d/mysql restart

# in host
docker stop wandb-local
docker run -e LOCAL_RESTORE=true --rm -d -v wandb:/vol -p 8080:8080 --name wandb-local wandb/local

and trigger the login or signup page http://localhost:8080/logout and click Login, input the created user in step 3, and it will bring me to the signup page. Now, input the Fullname and Username (same as created before test), click Continue, I can login. Wow, wow...

image

vanpelt commented 2 years ago

Hey @ericxsun we're working on a fix for this in the next release. You should never under any circumstances manually alter database records or create entities, that will potentially get you into an even more funky state but I'm glad you got in and apologies for this bug.

better629 commented 2 years ago

@ericxsun follow your steps, but still can't login in.sign up with pre-create account, but still occurs with entity already exists for the user xxxx. But can login in with restore account.

ericxsun commented 2 years ago

@better629 with the latest try(https://github.com/wandb/local/issues/64#issuecomment-1106030481), I do not meet such same error(entity already exists for the user xxxx). Does the username (xxxx) in signup page is same as the pre-create account xxxx@email-address? If same, I think you can show the infos in wandb_local.entities, remove it and retry(or re-do from the begining). Otherwise, try with the same username. Hope you could login.

better629 commented 2 years ago

@ericxsun I use the email with other username which I update the sql table. If it should be the same, I will try it. After login in, do u meet the backend api problem when save the license?

ericxsun commented 2 years ago

@ericxsun I use the email with other username which I update the sql table. If it should be the same, I will try it. After login in, do u meet the backend api problem when save the license?

@better629

1）Yes, I think the username and email-name should be same.

2）No, I didn't meet the error(the backend api problem). All is okay:

image

Or you can start the wandb-local with LICENSE (I mean, do not update LICENSE through Add LICENSE): docker run -e LICENSE=your-LICENSE --rm -d -v wandb:/vol -p 8080:8080 --name wandb-local wandb/local. It's okay.

image
Jeffrey28 commented 2 years ago
image

There has been no change in this interface. Is there any method to tackle it?

ericxsun commented 2 years ago
image

There has been no change in this interface. Is there any method to tackle it?

It'll cost much time to init the environment

Waiting!!!

or try again(if it cost too much time) :

docker stop wandb-local
docker volume rm wandb

docker run -e LOCAL_RESTORE=true --rm -d -v wandb:/vol -p 8080:8080 --name wandb-local wandb/local

Wish you good luck

Jeffrey28 commented 2 years ago

I have tried and waited for a long time almost one day. It still doesn't work.

vanpelt commented 2 years ago

@Jeffrey28 can you share the output of docker logs wandb-local

Jeffrey28 commented 2 years ago

Here it comes!


*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/01_enable-services.sh...
*** Copying services to runit
*** Copying jobber template
*** Enabling restore mode

Login at http://localhost:8080/api/login username: restore@wandb.com password: Ay56WFwuEPXTBA==

Enabling production mode Running /etc/my_init.d/02_load-settings.sh... Loading settings... Booting runit daemon... Runit started as PID 46 Starting wandb servers... Minio still starting up... Migrating database... panic: Dirty database version 2. Fix and force version.

goroutine 1 [running]: main.main() /mnt/ramdisk/core/services/gorilla/cmd/migrate/main.go:125 +0x8d6 All services started Access W&B at http://localhost:8080 !!! Unable to start backend, see logs at /var/logs. Trying again in 1 minute !!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console Starting wandb servers... All services started Access W&B at http://localhost:8080 Starting wandb servers... All services started Access W&B at http://localhost:8080 !!! Unable to start backend, see logs at /var/logs. Trying again in 1 minute !!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console

Jeffrey28 commented 2 years ago

And there is no a file named logs in the folder /var.

Jeffrey28 commented 2 years ago 
*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/01_enable-services.sh...
*** Copying services to runit
*** Copying jobber template
*** Enabling restore mode

Login at http://localhost:8080/api/login
  username: restore@wandb.com
  password: fJqL0uj1fewDqg==

*** Enabling production mode
*** Running /etc/my_init.d/02_load-settings.sh...
*** Loading settings...
2022/05/18 02:59:40 Created default user
2022/05/18 02:59:40 Generating new session key for auth...
2022/05/18 02:59:40 Generating new certificate and key for auth...
*** Booting runit daemon...
*** Runit started as PID 49
*** Setting up mysql database...
*** Starting wandb servers...
*** Minio still starting up...
*** Configuring minio...
Bucket created successfully `local/local-files`.
*** Minio still starting up...
Successfully added arn:minio:sqs:wandb-local:_:redis
*** Migrating database...
[mysql] 2022/05/18 03:04:00 packets.go:37: read tcp 127.0.0.1:60638->127.0.0.1:3306: i/o timeout
[mysql] 2022/05/18 03:04:00 connection.go:299: invalid connection
panic: 2 errors occurred:
    * migration failed in line 0: ALTER TABLE runs ADD COLUMN deleted_at datetime DEFAULT NULL;
 (details: invalid connection)
    * driver: bad connection in line 0: SELECT RELEASE_LOCK(?)

goroutine 1 [running]:
main.main()
    /mnt/ramdisk/core/services/gorilla/cmd/migrate/main.go:125 +0x8d6
*** All services started
*** Access W&B at http://localhost:8080
*** Starting wandb servers...
*** All services started
*** Access W&B at http://localhost:8080
!!! Unable to start backend, see logs at /var/logs.  Trying again in 1 minute
!!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console
vanpelt commented 2 years ago

The database is timing out when migrating. There's an issue with your docker filesystem driver. Can you provide answers to:

  1. Why are you running a local server?
  2. What OS and docker version are you using?
Jeffrey28 commented 2 years ago
  1. Running a local server is useful when the network is poor. I often encounter data cannot be uploaded and network errors.
  2. Ubuntu 20.04, Docker version 20.10.16
vanpelt commented 2 years ago

Hmm, let's try the following and see if things come up:

docker stop wandb-local docker rm volume wandb docker run -d --rm -v wandb:/vol/env:delegated -p 8080:8080 --name wandb-local wandb/local:latest docker logs -f wandb-local

Hopefully we don't see any errors in those logs and everything comes up after ~5 minutes or so. The main change here is adding the :delegated flag to the volume mount. That should improve disk performance.

Jeffrey28 commented 2 years ago

Hmm, let's try the following and see if things come up:

docker stop wandb-local docker rm volume wandb docker run -d --rm -v wandb:/vol/env:delegated -p 8080:8080 --name wandb-local wandb/local:latest docker logs -f wandb-local

Hopefully we don't see any errors in those logs and everything comes up after ~5 minutes or so. The main change here is adding the :delegated flag to the volume mount. That should improve disk performance.

Still same errors. Running /etc/my_init.d/00_regen_ssh_host_keys.sh... Running /etc/my_init.d/01_enable-services.sh... Copying services to runit Copying jobber template Enabling production mode Running /etc/my_init.d/02load-settings.sh... Loading settings... 2022/05/19 00:19:09 Created default user 2022/05/19 00:19:09 Generating new session key for auth... 2022/05/19 00:19:09 Generating new certificate and key for auth... Booting runit daemon... Runit started as PID 987 Setting up mysql database... Starting wandb servers... Configuring minio... Bucket created successfully local/local-files. Successfully added arn:minio:sqs:wandb-local::redis *** Migrating database... [mysql] 2022/05/19 00:23:30 packets.go:37: read tcp 127.0.0.1:60560->127.0.0.1:3306: i/o timeout [mysql] 2022/05/19 00:23:30 connection.go:299: invalid connection panic: 2 errors occurred:

goroutine 1 [running]: main.main() /mnt/ramdisk/core/services/gorilla/cmd/migrate/main.go:125 +0x8d6 All services started Access W&B at http://localhost:8080 Starting wandb servers... All services started Access W&B at http://localhost:8080 !!! Unable to start backend, see logs at /var/logs. Trying again in 1 minute !!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console Starting wandb servers... All services started Access W&B at http://localhost:8080 Starting wandb servers... All services started Access W&B at http://localhost:8080 !!! Unable to start backend, see logs at /var/logs. Trying again in 1 minute !!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console Starting wandb servers... All services started Access W&B at http://localhost:8080 Starting wandb servers... All services started *** Access W&B at http://localhost:8080 !!! Unable to start backend, see logs at /var/logs. Trying again in 1 minute !!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console

vanpelt commented 2 years ago

Can you share the mysql logs?

docker exec -it wandb-local bash cat /var/logs/mysql.log

Jeffrey28 commented 2 years ago 
wandb@e37bc2627b9b:~$ cat /var/logs/mysql.log
cat: /var/logs/mysql.log: No such file or directory
wandb@e37bc2627b9b:~$
exalate-issue-sync[bot] commented 2 years ago

WandB Internal User commented: better629 commented: @ericxsun follow your steps, but still can't login in.sign up with pre-create account, but still occurs with entity already exists for the user xxxx. But can login in with restore account.

exalate-issue-sync[bot] commented 2 years ago

WandB Internal User commented: ericxsun commented:

@better629 with the latest try(https://github.com/wandb/local/issues/64#issuecomment-1106030481), I do not meet such same error(entity already exists for the user xxxx). Does the username (xxxx) in signup page is same as the pre-create account xxxx@email-address? If same, I think you can show the infos in wandb_local.entities, remove it and retry(or re-do from the begining). Otherwise, try with the same username. Hope you could login.

exalate-issue-sync[bot] commented 2 years ago

WandB Internal User commented: better629 commented: @ericxsun I use the email with other username which I update the sql table. If it should be the same, I will try it. After login in, do u meet the backend api problem when save the license?

exalate-issue-sync[bot] commented 2 years ago

WandB Internal User commented: Jeffrey28 commented:

image

There has been no change in this interface. Is there any method to tackle it?

exalate-issue-sync[bot] commented 2 years ago

WandB Internal User commented: Jeffrey28 commented: And there is no a file named logs in the folder /var.

exalate-issue-sync[bot] commented 2 years ago

WandB Internal User commented: Jeffrey28 commented: 

*** Running /etc/my_init.d/00_regen_ssh_host_keys.sh...
*** Running /etc/my_init.d/01_enable-services.sh...
*** Copying services to runit
*** Copying jobber template
*** Enabling restore mode

Login at http://localhost:8080/api/login
  username: restore@wandb.com
  password: fJqL0uj1fewDqg==

*** Enabling production mode
*** Running /etc/my_init.d/02_load-settings.sh...
*** Loading settings...
2022/05/18 02:59:40 Created default user
2022/05/18 02:59:40 Generating new session key for auth...
2022/05/18 02:59:40 Generating new certificate and key for auth...
*** Booting runit daemon...
*** Runit started as PID 49
*** Setting up mysql database...
*** Starting wandb servers...
*** Minio still starting up...
*** Configuring minio...
Bucket created successfully `local/local-files`.
*** Minio still starting up...
Successfully added arn:minio:sqs:wandb-local:_:redis
*** Migrating database...
[mysql] 2022/05/18 03:04:00 packets.go:37: read tcp 127.0.0.1:60638->127.0.0.1:3306: i/o timeout
[mysql] 2022/05/18 03:04:00 connection.go:299: invalid connection
panic: 2 errors occurred:
    * migration failed in line 0: ALTER TABLE runs ADD COLUMN deleted_at datetime DEFAULT NULL;
 (details: invalid connection)
    * driver: bad connection in line 0: SELECT RELEASE_LOCK(?)

goroutine 1 [running]:
main.main()
    /mnt/ramdisk/core/services/gorilla/cmd/migrate/main.go:125 +0x8d6
*** All services started
*** Access W&B at http://localhost:8080
*** Starting wandb servers...
*** All services started
*** Access W&B at http://localhost:8080
!!! Unable to start backend, see logs at /var/logs.  Trying again in 1 minute
!!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console
exalate-issue-sync[bot] commented 2 years ago

WandB Internal User commented: vanpelt commented: The database is timing out when migrating. There's an issue with your docker filesystem driver. Can you provide answers to:

  1. Why are you running a local server?
  2. What OS and docker version are you using?
exalate-issue-sync[bot] commented 2 years ago

WandB Internal User commented: vanpelt commented: Hmm, let's try the following and see if things come up:

docker stop wandb-local docker rm volume wandb docker run -d --rm -v wandb:/vol/env:delegated -p 8080:8080 --name wandb-local wandb/local:latest docker logs -f wandb-local

Hopefully we don't see any errors in those logs and everything comes up after ~5 minutes or so. The main change here is adding the :delegated flag to the volume mount. That should improve disk performance.

exalate-issue-sync[bot] commented 2 years ago

WandB Internal User commented: Jeffrey28 commented:

Hmm, let's try the following and see if things come up:

docker stop wandb-local docker rm volume wandb docker run -d --rm -v wandb:/vol/env:delegated -p 8080:8080 --name wandb-local wandb/local:latest docker logs -f wandb-local

Hopefully we don't see any errors in those logs and everything comes up after ~5 minutes or so. The main change here is adding the :delegated flag to the volume mount. That should improve disk performance.

Still same errors. Running /etc/my_init.d/00_regen_ssh_host_keys.sh... Running /etc/my_init.d/01_enable-services.sh... Copying services to runit Copying jobber template Enabling production mode Running /etc/my_init.d/02load-settings.sh... Loading settings... 2022/05/19 00:19:09 Created default user 2022/05/19 00:19:09 Generating new session key for auth... 2022/05/19 00:19:09 Generating new certificate and key for auth... Booting runit daemon... Runit started as PID 987 Setting up mysql database... Starting wandb servers... Configuring minio... Bucket created successfully local/local-files. Successfully added arn:minio:sqs:wandb-local::redis *** Migrating database... [mysql] 2022/05/19 00:23:30 packets.go:37: read tcp 127.0.0.1:60560->127.0.0.1:3306: i/o timeout [mysql] 2022/05/19 00:23:30 connection.go:299: invalid connection panic: 2 errors occurred:

goroutine 1 [running]: main.main() /mnt/ramdisk/core/services/gorilla/cmd/migrate/main.go:125 +0x8d6 All services started Access W&B at http://localhost:8080 Starting wandb servers... All services started Access W&B at http://localhost:8080 !!! Unable to start backend, see logs at /var/logs. Trying again in 1 minute !!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console Starting wandb servers... All services started Access W&B at http://localhost:8080 Starting wandb servers... All services started Access W&B at http://localhost:8080 !!! Unable to start backend, see logs at /var/logs. Trying again in 1 minute !!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console Starting wandb servers... All services started Access W&B at http://localhost:8080 Starting wandb servers... All services started *** Access W&B at http://localhost:8080 !!! Unable to start backend, see logs at /var/logs. Trying again in 1 minute !!! Set the environment variable LOCAL_RESTORE=true if you're unable to login to the admin console

exalate-issue-sync[bot] commented 2 years ago

WandB Internal User commented: Jeffrey28 commented: 

wandb@e37bc2627b9b:~$ cat /var/logs/mysql.log
cat: /var/logs/mysql.log: No such file or directory
wandb@e37bc2627b9b:~$
nate-wandb commented 2 years ago

Hi @ericxsun are there any log files within /var/logs/ and if so could you send the contents of those?

nate-wandb commented 2 years ago

@ericxsun I wanted to follow up on this and see if you still needed help with this?

ericxsun commented 2 years ago

@ericxsun I wanted to follow up on this and see if you still needed help with this?

No, thank you, I've fixed it. Please refer to https://github.com/wandb/local/issues/64#issuecomment-1106030481.

Guitaricet commented 2 years ago

I think wandb is amazing and I would really love to be able to use it locally, but this bug exists for at least 3 months now and haven't been resolved. I wasn't able to successfully run wandb inside a kubernetes environment after hours of looking at the logs and the github issues. I might be wrong, but at least to me this bug seems critical.

Guitaricet commented 2 years ago

UPD: it looks like this problem resolves if you use port 8080 to launch wandb server and to connect to it. E.g., port-forward 8080:8080. At least the docker version of wandb works for me (wandb version 0.12.21).

azarezade commented 1 year ago

I still have the same error. After login for first time and signup using wandb UI, I can see my profile, but if I logout then I can't login anymore and receive 401 Unauthorized error "error": "user not found". Here is my docker-compose.yml:

version: '3'
services:
  wandb-local:
    image: wandb/local
    container_name: wandb-local
    ports:
      - "8080:8080"
    volumes:
      - data:/vol
    environment:
      - HOST=http://<my-host>:8080
    restart: unless-stopped
volumes:
  data: