Workspace get stuck: Creation in progress... 100%

anibal-aguila commented 1 week ago

Hi, Using traefik and additional last image of workspace the instance never start..

bb@nixops-A1:~/dpl/now/huly/ > podman-compose ps
CONTAINER ID  IMAGE                                          COMMAND               CREATED         STATUS                   PORTS                             NAMES
fc7b2dbdcaa1  docker.io/library/mongo:7-jammy                mongod                24 minutes ago  Up 24 minutes            0.0.0.0:27017->27017/tcp          mongodb
3a32e5783b13  docker.io/minio/minio:latest                   server /data --ad...  24 minutes ago  Up 24 minutes            0.0.0.0:9000-9001->9000-9001/tcp  huly_minio_1
25d609ac4639  docker.io/library/elasticsearch:7.14.2         /bin/sh -c ./bin/...  24 minutes ago  Up 24 minutes (healthy)  0.0.0.0:9200->9200/tcp            huly_elastic_1
e0c185fa1e2c  docker.io/hardcoreeng/rekoni-service:v0.6.313  node ./bundle.js      24 minutes ago  Up 24 minutes            0.0.0.0:4004->4004/tcp            huly_rekoni_1
011885ab70c3  docker.io/hardcoreeng/account:v0.6.313         node bundle.js        24 minutes ago  Up 24 minutes            0.0.0.0:3000->3000/tcp            huly_account_1
b0185fdfaf9f  docker.io/hardcoreeng/workspace:v0.6.313       node bundle.js        24 minutes ago  Up 24 minutes                                              huly_workspace_1
d90bdaef429e  docker.io/hardcoreeng/transactor:v0.6.313      /bin/sh -c node b...  24 minutes ago  Up 24 minutes                                              huly_transactor_1
b99ecf7d17ce  docker.io/hardcoreeng/collaborator:v0.6.313    node bundle.js        24 minutes ago  Up 24 minutes                                              huly_collaborator_1
98331928498f  docker.io/hardcoreeng/front:v0.6.313           node ./bundle.js      24 minutes ago  Up 24 minutes            0.0.0.0:9010->8080/tcp            huly_front_1
bb@nixops-A1:~/dpl/now/huly/ > podman ps -a | grep traefik
7b3972836d81  docker.io/library/traefik:3.1.4                      traefik               3 hours ago     Up 3 hours               0.0.0.0:22->22/tcp, 0.0.0.0:80->80/tcp, 0.0.0.0:389->389/tcp, 0.0.0.0:443->443/tcp, 0.0.0.0:9011-9012->9011-9012/tcp, 0.0.0.0:37017->37017/tcp, 0.0.0.0:63306->63306/tcp, 0.0.0.0:65432->65432/tcp  traefik-front

Thanks in advance,

0xtejas commented 1 week ago

I have a similar issue. The deployment is stuck creating at 10%. I have deployed it on K8s.

jondef commented 1 week ago

Same but mine is stuck at 0%

0xtejas commented 1 week ago

I tried v0.6.313 releases for all components and it gets stuck at 100%

The upper half is workspace and the lower one is account.

MongoDB output

MinIO is created but empty

0xtejas commented 1 week ago

I checked the next day, and it has failed to create but the UI says creating.

FireflyHacker commented 1 week ago

I am having the same issue trying to run it on k8s. I tracked the error message down to line 203 of platform/server/workspace-service/src/service.ts and it seems to be from a failed fetch request from updateWorkspaceInfo on line 166. I don't know typescript well enough to get much further than that. It could be a set up issue or it could be a bug in the code, but something is happening at the fetch request.

muradbozik commented 6 days ago

@FireflyHacker I think update fails because of transactor pod. In workspace deployment the transactor url should be corrected. It should bews://transactor instead of ws://transactor:3333. Reason is the service is set up at port 80 by default for transactor. Or you can change the transactor service port to 3333 (same as the target port)

F04C commented 5 days ago

Stucked also at creation in progress... 100%

`services: mongodb: image: "mongo:4.4" container_name: mongodb environment:

PUID=${PUID}
PGID=${PGID} volumes:
db:/data/db ports:
27017:27017
restart: unless-stopped

minio: image: "minio/minio" command: server /data --address ":9000" --console-address ":9001" ports:
9020:9020
9021:9021
volumes:
files:/data restart: unless-stopped

elastic: image: "elasticsearch:7.14.2" command: | /bin/sh -c "./bin/elasticsearch-plugin list | grep -q ingest-attachment || yes | ./bin/elasticsearch-plugin install --silent ingest-attachment; /usr/local/bin/docker-entrypoint.sh eswrapper" volumes:
elastic:/usr/share/elasticsearch/data ports:
9200:9200
environment:
ELASTICSEARCH_PORT_NUMBER=${ELASTICSEARCH_PORT_NUMBER}
BITNAMI_DEBUG=${BITNAMI_DEBUG}
discovery.type=single-node
ES_JAVA_OPTS=${ES_JAVA_OPTS}
http.cors.enabled=${CORS_ENABLED}
http.cors.allow-origin=${CORS_ALLOW_ORIGIN} healthcheck: interval: 20s retries: 10 test: curl -s http://localhost:9200/_cluster/health | grep -vq '"status":"red"' restart: unless-stopped

account: image: hardcoreeng/account:${HULY_VERSION} links:
mongodb
minio ports:
3030:3001 environment:
SERVER_PORT=3030
SERVER_SECRET=${HULY_SECRET}
MONGO_URL=mongodb://mongodb:27017
DB_URL=mongodb://mongodb:27017
TRANSACTOR_URL=ws://transactor;ws://${SERVER_ADDRESS}:3333
STORAGE_CONFIG=minio|minio?accessKey=${MINIO_ACCESS_KEY}&secretKey=${MINIO_SECRET_KEY}
FRONT_URL=http://front:8080
MODEL_ENABLED=${MODEL_ENABLED}
ACCOUNTS_URL=http://${SERVER_ADDRESS}:3001
ACCOUNT_PORT=${ACCOUNT_PORT} restart: unless-stopped

workspace: image: hardcoreeng/workspace:${HULY_VERSION} links:
mongodb
minio environment:
SERVER_SECRET=${HULY_SECRET}
MONGO_URL=mongodb://mongodb:27017
DB_URL=mongodb://mongodb:27017
TRANSACTOR_URL=ws://transactor:3334
STORAGE_CONFIG=minio|minio?accessKey=${MINIO_ACCESS_KEY}&secretKey=${MINIO_SECRET_KEY}
MODEL_ENABLED=${MODEL_ENABLED}
ACCOUNTS_URL=http://account:3001
NOTIFY_INBOX_ONLY=${NOTIFY_INBOX_ONLY} restart: unless-stopped

front: image: hardcoreeng/front:${HULY_VERSION} links:
mongodb
minio
elastic
collaborator
transactor ports:
8100:8087 environment:
SERVER_PORT=8087
SERVER_SECRET=${HULY_SECRET}
ACCOUNTS_URL=http://${SERVER_ADDRESS}:3030
REKONI_URL=http://${SERVER_ADDRESS}:4005
CALENDAR_URL=http://${SERVER_ADDRESS}:8095
GMAIL_URL=http://${SERVER_ADDRESS}:8088
TELEGRAM_URL=http://${SERVER_ADDRESS}:8086
UPLOAD_URL=/files
ELASTIC_URL=http://elastic:9201 # Updated port
COLLABORATOR_URL=ws://${SERVER_ADDRESS}:3080 # Changed from 3078 to 3079
STORAGE_CONFIG=minio|minio?accessKey=${MINIO_ACCESS_KEY}&secretKey=${MINIO_SECRET_KEY}
MONGO_URL=mongodb://mongodb:27017
TITLE=${TITLE}
DEFAULT_LANGUAGE=${DEFAULT_LANGUAGE}
LAST_NAME_FIRST=${LAST_NAME_FIRST} restart: unless-stopped

collaborator: image: hardcoreeng/collaborator:${HULY_VERSION} links:
mongodb
minio
transactor ports:
3080:3080 # Changed from 3078 to 3079 environment:
COLLABORATOR_PORT=${COLLABORATOR_PORT} # Updated port
SECRET=${HULY_SECRET}
ACCOUNTS_URL=http://account:3001
MONGO_URL=mongodb://mongodb:27017
STORAGE_CONFIG=minio|minio?accessKey=${MINIO_ACCESS_KEY}&secretKey=${MINIO_SECRET_KEY} restart: unless-stopped

transactor: image: hardcoreeng/transactor:${HULY_VERSION} links:
mongodb
elastic
minio
account ports:
3334:3334 environment:
SERVER_PORT=3334
SERVER_SECRET=${HULY_SECRET}
SERVER_CURSOR_MAXTIMEMS=30000
ELASTIC_URL=http://elastic:9201
ELASTIC_INDEX_NAME=${ELASTIC_INDEX_NAME}
MONGO_URL=mongodb://mongodb:27017
DB_URL=mongodb://mongodb:27017
METRICS_CONSOLE=${METRICS_CONSOLE}
METRICS_FILE=${METRICS_FILE}
STORAGE_CONFIG=minio|minio?accessKey=${MINIO_ACCESS_KEY}&secretKey=${MINIO_SECRET_KEY}
REKONI_URL=http://rekoni:4004
FRONT_URL=http://${SERVER_ADDRESS}:8088
ACCOUNTS_URL=http://account:3001
LAST_NAME_FIRST=${LAST_NAME_FIRST} restart: unless-stopped

rekoni: image: hardcoreeng/rekoni-service:${HULY_VERSION} ports:
4004:4004
environment:
SECRET=${HULY_SECRET} deploy: resources: limits: memory: 500M restart: unless-stopped

volumes: db: files: elastic: etcd:`

{ "result": [ { "workspace": "test222222222", "workspaceUrl": "test222222222", "version": { "major": 0, "minor": 0, "patch": 0 }, "branding": "huly", "workspaceName": "test222222222", "disabled": true, "region": "", "mode": "pending-creation", "progress": 0, "createdOn": 1729005506907, "lastVisit": 1729005567288, "createdBy": "test2", "lastProcessingTime": 0, "attempts": 0, "workspaceId": "w-test2-test22222222-670e87c2-59534b17fb-ca746a" } ] }

here is the netstat of the workspace

Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 127.0.0.11:33615 0.0.0.0:* LISTEN tcp6 0 0 :::3334 :::* LISTEN udp 0 0 127.0.0.11:59903 0.0.0.0:*

muradbozik commented 5 days ago

@F04C You are using SERVICE_PORT=3334 for transactor. In the account part you need to set TRANSACTOR_URL=ws://transactor:3334;ws://${SERVER_ADDRESS}:3334 similar to workspace.

F04C commented 5 days ago

Thank you ❤️

FireflyHacker commented 3 days ago

@muradbozik You were right. I tried changing everything from service.huly.example to service.huly.domain.com in the yaml files but that did not work. Changing all the ingress files to just the name of the service and changing the config to just the external IP from my load balancer seems to have resolved any issues (at least for v0.6.325). Oh and I had to add a postgres container to the k8s deployment.

Thank you for the advice!

hcengineering / huly-selfhost

Workspace get stuck: Creation in progress... 100% #45