How can I get more information to help me deploy tapir?

WeiMengXS commented 8 months ago

When I try to upload the provider, it returns {"errorId":"41ede21f-72c9-4df3-a33b-e9f75eea3965","errors":[{"message":"An unexpected error has occurred. Please raise an issue if you think this is a bug."}] But based on the existing documents, I haven't been able to find out where the problem lies?

Console output of tapir：

No further information...

PacoVK commented 8 months ago

Thanks for reporting! Could you please share further insights? Ideally the request you made and the folder structure of your provider, if you tried to push a custom build one.

WeiMengXS commented 8 months ago

Hello paco! Can you help me answer this question?

I got this error while uploading. 2023-12-25 10:10:16,416 SEVERE [api.map.exc.ThrowableMapper] (executor-thread-3) errorId e8cc5635-4546-4f64-8e88-b816564585a2: java.lang.RuntimeException: java.nio.file.NoSuchFileException: /tmp/13909245068916160045/archive/terraform-provider-yunti_0.0.2_darwin_arm64.zip

I entered the container according to the path and found that the file looked like this:

Is there something wrong with my operation?

The directory structure I uploaded is like this:

Commands used: curl -XPOST --fail-with-body -H 'x-api-key: xxxxxxxx' -F archive=@archive.zip "https://example.com/terraform/providers/v1/fc/yunti/v0.0.2"

PacoVK commented 8 months ago

Assuming that the archive size is 318804492 bytes (~319MB), did you already set API_MAX_BODY_SIZE to a custom value? Because default is 100MB.

WeiMengXS commented 8 months ago

Hello, I have set API_MAX_BODY_SIZE to 1GB, but I'm still encountering the same issue even though the file size being uploaded this time is around 50MB. -e API_MAX_BODY_SIZE=1024M

Commands used: curl -XPOST --fail-with-body -H 'x-api-key: aZAk6Y6Tu8PhLmcYZnaUN7M7' -F archive=@archive.zip "https://example.com/terraform/providers/v1/fc/yunti/v0.0.2" Run:

docker run -d --name tapir   -p 8088:8080  
-e API_MAX_BODY_SIZE=1024M 
-e BACKEND_CONFIG=elasticsearch   
-e BACKEND_ELASTICSEARCH_HOST=xxxx:9200  
 -e STORAGE_CONFIG=host   
-e REGISTRY_HOSTNAME=localhost  
 -e REGISTRY_PORT=443 
-e REGISTRY_GPG_KEYS_0__ID=xxxx 
-e REGISTRY_GPG_KEYS_0__ASCII_ARMOR=xxxxx 
-e AUTH_ENDPOINT=http://xxxxx:8080/auth/realms/terraform 
-e AUTH_CLIENT_ID=registry 
pacovk/tapir

PacoVK commented 8 months ago

Ah I think the docs could be more expressive, for the STORAGE_CONFIG. You set STORAGE_CONFIG=host which is not valid, to use local volume please set to STORAGE_CONFIG=local

Let me know if that fixes the issue, I will extend the startup script to also check for the storage config and print a better Error message in case it is invalid.

WeiMengXS commented 8 months ago

Thank you for your reply. Oh, sorry for the mistake. However, I have already set STORAGE_CONFIG to local, but the issue still persists. Are there any other directions for troubleshooting?

 docker run -d --name tapir   -p 8088:8080  
-e API_MAX_BODY_SIZE=400M 
-e BACKEND_CONFIG=elasticsearch   
-e BACKEND_ELASTICSEARCH_HOST=xxx:9200   
-e STORAGE_CONFIG=local   
-e REGISTRY_HOSTNAME=localhost   
-e REGISTRY_PORT=443 
-e REGISTRY_GPG_KEYS_0__ID=xxx 
-e REGISTRY_GPG_KEYS_0__ASCII_ARMOR=xxx 
-e AUTH_ENDPOINT=http://xxxx:8080/auth/realms/terraform 
-e AUTH_CLIENT_ID=registry 
pacovk/tapir

Commands used:

$ curl -XPOST  -H 'x-api-key:aZAk6Y6Tu8PhLmcYZnaUN7M7' --fail-with-body  -F archive=@archive.zip https://examlpe.cn/terraform/providers/v1/fc/yunti/v0.0.2
curl: (22) The requested URL returned error: 500
{"errorId":"53b38893-a350-4027-b1ba-81a3a9894435","errors":[{"message":"An unexpected error has occurred. Please raise an issue if you think this is a bug."}]

My file:

The backend files are:

PacoVK commented 8 months ago

I will try to reproduce and investigate a bit further. Thanks for the comprehensive report👌

PacoVK commented 8 months ago

@WeiMengXS i tried to write a reproducer with the following: ℹ️ For Keycloak i used my dev setup, but that should not have any influence, since you are already able to login and create the deploykey. ℹ️ I also did no setup any GPG_ things because you wrote the error occures during upload of the provider. I omitted any setting that is relevant for downloading providers to keep it simple

docker-compose.yml

version: '3'
services:

  tapir:
    image: pacovk/tapir
    ports:
      - "8088:8080"
    environment:
      API_MAX_BODY_SIZE: 400M
      BACKEND_CONFIG: elasticsearch
      BACKEND_ELASTICSEARCH_HOST: opensearch-node1:9200
      STORAGE_CONFIG: local
      AUTH_ENDPOINT: http://keycloak:8080/realms/tapir

  keycloak:
    image: quay.io/keycloak/keycloak:22.0
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./dev/scripts/keycloak:/opt/keycloak/data/import
    ports:
      - "8080:8080"
    command:
      - start-dev --import-realm
    environment:
      - KEYCLOAK_ADMIN=admin
      - KEYCLOAK_ADMIN_PASSWORD=admin

  opensearch-node1:
    image: opensearchproject/opensearch
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - plugins.security.disabled=true
      - node.name=opensearch-node1 # Name the node that will run in this container
      - discovery.type=single-node
      - bootstrap.memory_lock=true # Disable JVM heap memory swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
    ulimits:
      memlock:
        soft: -1 # Set memlock to unlimited (no soft or hard limit)
        hard: -1
      nofile:
        soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
        hard: 65536
    ports:
      - "9200:9200" # REST API
      - "9600:9600" # Performance Analyzer

I used a fake provider as demo with similar folder structure as you described I did then the following:

create a deploykey via Tapir for the provider, where namespace=fc and type=yunti
issue a upload command to Tapir via curl curl -XPOST -H 'x-api-key:tM6pX3IlTBROmLAJk8Nm4Fhy' --fail-with-body -F archive=@Archiv.zip http://localhost:8088/terraform/providers/v1/fc/yunti/v0.0.2
No error...

I used the following fake provider source: Archiv.zip

How to proceed?

verify you used the latest Tapir version (to be sure, run docker pull pacovk/tapir)
Increase the log level by setting envinroment variable QUARKUS_LOG_LEVEL=debug
Please check if you can upload the "fake provider" i used here.
If not, please also check if you have any extra setting for Tapir set in your container, apart from those you already posted.
For further investigation, please try to setup a local Tapir instance from the docker compose reference aforementioned and try to upload your provider there (mind without SSL and GPG you wont be able to download but it should at least work with the upload
Keep me posted here, in case i can further assist i am happy to do so

WeiMengXS commented 8 months ago

Thank you very much for this detailed guidance. I have been able to successfully deploy and publish my provider!

WeiMengXS commented 8 months ago

Initializing the backend...

Initializing provider plugins...

Finding latest version of xxx.com.ngrok-free.app/fc/yunti...
Installing xxx.com/fc/yunti v0.0.2... ╷ │ Error: Failed to install provider │ │ Error while installing xxx.com/fc/yunti v0.0.2: checksum list has no SHA-256 hash for "terraform-provider-yunti_0.0.2_darwin_arm64.zip" ╵ But I encountered this problem. I debugged and found that it seemed to be an auth problem with keylocak.

PacoVK commented 8 months ago

I think that is more about the provider artefact. You strictly have to fulfill the requirements described in the official Hashicorp docs

In your case the SHASUM file might not be correct. Read here if you try to manually prepare the release. The GPG key used for signing must be the same passed to Tapir!

WeiMengXS commented 8 months ago

Hashicorp 官方文档I am quite familiar with it now, it seems that when TF is accessing the backend, it is redirected to the Keycloak's verification URI. Do you have any reference for setting up Keycloak? Because I am not very familiar with Keycloak itself, This is a bit confusing for me.

"_NOTE: To use Tapir UI you need to be authenticated. However, you can read the registry without authentication. In particular the Terraform CLI will work without authentication" In other words, how to implement this.

PacoVK commented 8 months ago

You found a bug with the local storage backend, I will try to fix it soon. The storage api needs to be accessible without authentication to allow read from the registry! No issue with Keycloak. Thanks for sharing this!

WeiMengXS commented 8 months ago

When is it expected to be restored? I'm very interested in this project!

PacoVK commented 8 months ago

I already fixed it and will create a release probably today, latest tomorow.

PacoVK commented 8 months ago

Here we go 0.6.2 contains the fix https://github.com/PacoVK/tapir/releases/tag/0.6.2

WeiMengXS commented 8 months ago

I have verified it and there seems to be no issue. Thank you very much. However, it seems that there could be room for improvement in displaying it on the frontend,I'm not sure if there is a misconfiguration somewhere.

If this is a domain name, should the 'https://' prefix actually be removed? In Terraform, it is expected to follow the format [hostname/][namespace/]name. The "source" attribute must be in the format "[hostname/][namespace/]name"

PacoVK commented 8 months ago

Good point, I'll add that on the list. Actually it makes no difference but without the protocol it follows the naming of hashicorp in a consistent way 👌 Thanks again for the valuable feedback!

PacoVK commented 8 months ago

@all-contributors please add @WeiMengXS for bug, ideas

allcontributors[bot] commented 8 months ago

@PacoVK

I've put up a pull request to add @WeiMengXS! :tada:

PacoVK / tapir

How can I get more information to help me deploy tapir? #360

How to proceed?