Setting up a private Open VSX registry

aallrd commented 1 year ago

Hello,

Thank you for this project.

I am trying to setup an internal Open VSX registry for my development team in order to be able to publish our private extensions for VSCode. I read the deployment documentation but I am struggling to configure the openvsx-server. I understand it requires an application.yml file mounted at /home/openvsx/server/config/application.yml in the container, but I cannot find any basic template to get started from. I have found this application.yml from open-vsx.org but it seems very complex for what I am trying to setup. My goal is to have a publicly available internal instance with local storage, basically the simplest deployment possible for our small team. Could you share some documentation/material to be able to setup this deployment?

Thank you.

amvanbaren commented 1 year ago

Hi @aallrd, thank you for choosing Open VSX. I think the open-vsx.org application.yml file is a good base to start from. There are a couple changes you'll have to make to the file:

remove spring.config.import
remove ovsx.storage.primary-service, because you want to use local storage
remove the ovsx.eclipse section
remove ovsx.publishing.require-license
remove ovsx.request.duration.threshold
if you only intend to run one instance/pod, you can remove ovsx.migrations.delay.seconds
if you don't need rate-limiting, you can set bucket4j.enabled to false and remove all the filters

Hope this helps. Please report back your findings.

aallrd commented 1 year ago

Hello, Thank you for the help. I reduced the application.yml following your recommendations and removed the security.oauth2 section as well:

server:
  address: 0.0.0.0
  port: 8080
  shutdown: graceful
  compression:
    enabled: true
    mime-types: text/html,text/plain,text/css,application/javascript,application/json,text/xml,application/xml,application/xml+rss,text/javascript
    min-response-size: 1024
spring:
  cache:
    jcache:
      config: classpath:ehcache.xml
  datasource:
    hikari:
      maximum-pool-size: 10
      leak-detection-threshold: 30000
  flyway:
    baseline-on-migrate: false
  jpa:
    open-in-view: false
    properties:
      hibernate:
        dialect: org.hibernate.dialect.PostgreSQLDialect
  resources:
    cache:
      cachecontrol:
        max-age: 4h
        cache-public: true
  session:
    store-type: jdbc
  lifecycle:
    timeout-per-shutdown-phase: 10s
management:
  server:
    port: 8081
  health:
    probes:
      enabled: true
org:
  jobrunr:
    job-scheduler:
      enabled: true
    background-job-server:
      enabled: true
      worker-count: 2
    dashboard:
      enabled: false
    database:
      type: sql
    miscellaneous:
      allow-anonymous-data-usage: false
bucket4j:
  enabled: false
ovsx:
  webui:
    frontendRoutes: "/extension/**,/namespace/**,/user-settings/**,/admin-dashboard/**,/about,/publisher-agreement-*,/terms-of-use"
  elasticsearch:
    enabled: true
    ssl: true
    relevance:
      rating: 0.2
      downloads: 1.0
      timestamp: 3.0

I start the server loading this file using the below command line:

docker run --rm -it -v $(pwd)/application.yml:/home/openvsx/server/config/application.yml ghcr.io/eclipse/openvsx-server:v0.9.4

It throws the below database related error:

***************************
APPLICATION FAILED TO START
***************************

Description:

Failed to configure a DataSource: 'url' attribute is not specified and no embedded datasource could be configured.

Reason: Failed to determine a suitable driver class

Action:

Consider the following:
        If you want an embedded database (H2, HSQL or Derby), please put it on the classpath.
        If you have database settings to be loaded from a particular profile you may need to activate it (no profiles are currently active).

I have tried modifying the spring.datasource property to use one of the proposed embedded database but I was not able to succeed (I think drivers are missing).

I am guessing otherwise I need to setup an external PostgreSQL database and fill the configuration under spring.datasource, but I am not sure how to do that and what to do about the spring.datasource.hikari thing.

Do you have any idea?

aallrd commented 1 year ago

To move forward in the meantime I have managed to setup an external PostgreSQL database:

docker run -d -p 5432:5432 --name postgres-openvsx -e POSTGRES_PASSWORD=postgres postgres:14.7-alpine
psql -h localhost -p 5432 -U postgres

I added the below properties in spring.datasource and kept the spring.datasource.hikari ones:

  datasource:
    url: jdbc:postgresql://0.0.0.0:5432/
    username: postgres
    password: postgres
    hikari:
      maximum-pool-size: 10
      leak-detection-threshold: 30000

I started the server:

docker run --rm -it --network=host -v $(pwd)/application.yml:/home/openvsx/server/config/application.yml ghcr.io/eclipse/openvsx-server:v0.9.4

And it now fails because of the security.oauth2 property I removed earlier:

***************************
APPLICATION FAILED TO START
***************************

Description:

Field clientRegistrationRepository in org.eclipse.openvsx.security.TokenService required a bean of type 'org.springframework.security.oauth2.client.registration.ClientRegistrationRepository' that could not be found.

The injection point has the following annotations:
        - @org.springframework.beans.factory.annotation.Autowired(required=true)

Action:

Consider defining a bean of type 'org.springframework.security.oauth2.client.registration.ClientRegistrationRepository' in your configuration.

What's the simplest thing I could add there?

amvanbaren commented 1 year ago

Hi @aallrd, you need to setup a GitHub OAuth app and add the client-id and client-secret to the application.yml file.

spring:
  security:
    oauth2:
      client:
        registration:
          github:
            client-id: <CLIENT_ID>
            client-secret: <CLIENT_SECRET>

You also need to run a ElasticSearch 7.11.0 instance and configure it:

ovsx:
  elasticsearch:
    host: localhost:9200

aallrd commented 1 year ago

Can’t I just disable the authentication entirely ? This registry will be in an internal network without access to internet in the end.

amvanbaren commented 1 year ago

Not at this moment, see #337.

I think you can set random values for the github client-id and client-secret, just so that the server starts without throwing an exception.

You can then publish extensions using super_token, the super_user token. However, the webui login won't work. This means you can't access the user settings and admin dashboard.

aallrd commented 1 year ago

I am having a hard time connecting the openvsx-server to my elasticsearch instance.

Here is my docker-compose.yml:

version: "3.9"
services:
  openvsx-server:
    image: ghcr.io/eclipse/openvsx-server:v0.9.4
    ports:
      - "8080:8080"
    depends_on:
      elasticsearch:
        condition: service_healthy
      postgres:
        condition: service_healthy
  elasticsearch:
    image: elasticsearch:8.6.2
    environment:
      - xpack.security.enabled=false
      - discovery.type=single-node
    ports:
      - "9200:9200"
      - "9300:9300"
    healthcheck:
      test: ["CMD-SHELL", "curl http://elasticsearch:9200/_cluster/health?wait_for_status=green"]
      timeout: 60s
  postgres:
    image: postgres:14.7-alpine
    environment:
      - POSTGRES_PASSWORD=postgres
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 10s
      retries: 6
networks:
  default:
    name: openvsx-network

And my application.yml:

server:
  address: 0.0.0.0
  port: 8080
  shutdown: graceful
  compression:
    enabled: true
    mime-types: text/html,text/plain,text/css,application/javascript,application/json,text/xml,application/xml,application/xml+rss,text/javascript
    min-response-size: 1024
spring:
  cache:
    jcache:
      config: classpath:ehcache.xml
  datasource:
    url: jdbc:postgresql://postgres:5432/
    username: postgres
    password: postgres
    hikari:
      maximum-pool-size: 10
      leak-detection-threshold: 30000
  flyway:
    baseline-on-migrate: false
  jpa:
    open-in-view: false
    properties:
      hibernate:
        dialect: org.hibernate.dialect.PostgreSQLDialect
  resources:
    cache:
      cachecontrol:
        max-age: 4h
        cache-public: true
  session:
    store-type: jdbc
  lifecycle:
    timeout-per-shutdown-phase: 10s
  security:
    oauth2:
      client:
        registration:
          github:
            client-id: dummy
            client-secret: dummy
management:
  server:
    port: 8081
  health:
    probes:
      enabled: true
org:
  jobrunr:
    job-scheduler:
      enabled: true
    background-job-server:
      enabled: true
      worker-count: 2
    dashboard:
      enabled: false
    database:
      type: sql
    miscellaneous:
      allow-anonymous-data-usage: false
bucket4j:
  enabled: false
ovsx:
  webui:
    frontendRoutes: "/extension/**,/namespace/**,/user-settings/**,/admin-dashboard/**,/about,/publisher-agreement-*,/terms-of-use"
  elasticsearch:
    host: elasticsearch:9200
    enabled: true
    ssl: false
    relevance:
      rating: 0.2
      downloads: 1.0
      timestamp: 3.0

The openvsx-server waits for the elasticsearch service to be up as instructed:

$ curl -s http://0.0.0.0:9200/_cluster/health | jq
{
  "cluster_name": "docker-cluster",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 1,
  "number_of_data_nodes": 1,
  "active_primary_shards": 1,
  "active_shards": 1,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100
}

But I keep getting these errors:

ovsx-openvsx-server-1  | 2023-03-27 10:54:50.829  WARN 1 --- [           main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'recurringJobPostProcessor' defined in class path resource [org/jobrunr/spring/autoconfigure/JobRunrAutoConfiguration.class]: Unsatisfied dependency expressed through method 'recurringJobPostProcessor' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'jobScheduler' defined in class path resource [org/jobrunr/spring/autoconfigure/JobRunrAutoConfiguration.class]: Unsatisfied dependency expressed through method 'jobScheduler' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'storageProvider' defined in class path resource [org/jobrunr/spring/autoconfigure/storage/JobRunrElasticSearchStorageAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.jobrunr.storage.StorageProvider]: Factory method 'elasticSearchStorageProvider' threw exception; nested exception is ElasticsearchException[java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused]; nested: ExecutionException[java.net.ConnectException: Connection refused]; nested: ConnectException[Connection refused];
ovsx-openvsx-server-1  | 2023-03-27 10:54:50.887  INFO 1 --- [           main] ConditionEvaluationReportLoggingListener :
ovsx-openvsx-server-1  |
ovsx-openvsx-server-1  | Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
ovsx-openvsx-server-1  | 2023-03-27 10:54:50.936 ERROR 1 --- [           main] o.s.boot.SpringApplication               : Application run failed
ovsx-openvsx-server-1  |
ovsx-openvsx-server-1  | org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'recurringJobPostProcessor' defined in class path resource [org/jobrunr/spring/autoconfigure/JobRunrAutoConfiguration.class]: Unsatisfied dependency expressed through method 'recurringJobPostProcessor' parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'jobScheduler' defined in class path resource [org/jobrunr/spring/autoconfigure/JobRunrAutoConfiguration.class]: Unsatisfied dependency expressed through method 'jobScheduler' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'storageProvider' defined in class path resource [org/jobrunr/spring/autoconfigure/storage/JobRunrElasticSearchStorageAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.jobrunr.storage.StorageProvider]: Factory method 'elasticSearchStorageProvider' threw exception; nested exception is ElasticsearchException[java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused]; nested: ExecutionException[java.net.ConnectException: Connection refused]; nested: ConnectException[Connection refused];
ovsx-openvsx-server-1  |        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:800) ~[spring-beans-5.3.21.jar:5.3.21]
ovsx-openvsx-server-1  |        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:541) ~[spring-beans-5.3.21.jar:5.3.21]
ovsx-openvsx-server-1  |        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352) ~[spring-beans-5.3.21.jar:5.3.21]
ovsx-openvsx-server-1  |        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195) ~[spring-beans-5.3.21.jar:5.3.21]
ovsx-openvsx-server-1  |        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582) ~[spring-beans-5.3.21.jar:5.3.21]
ovsx-openvsx-server-1  |        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542) ~[spring-beans-5.3.21.jar:5.3.21]
ovsx-openvsx-server-1  |        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.21.jar:5.3.21]
ovsx-openvsx-server-1  |        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.21.jar:5.3.21]
ovsx-openvsx-server-1  |        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.21.jar:5.3.21]
ovsx-openvsx-server-1  |        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:213) ~[spring-beans-5.3.21.jar:5.3.21]
ovsx-openvsx-server-1  |        at org.springframework.context.support.PostProcessorRegistrationDelegate.registerBeanPostProcessors(PostProcessorRegistrationDelegate.java:270) ~[spring-context-5.3.21.jar:5.3.21]
ovsx-openvsx-server-1  |        at org.springframework.context.support.AbstractApplicationContext.registerBeanPostProcessors(AbstractApplicationContext.java:762) ~[spring-context-5.3.21.jar:5.3.21]
ovsx-openvsx-server-1  |        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:567) ~[spring-context-5.3.21.jar:5.3.21]
ovsx-openvsx-server-1  |        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:147) ~[spring-boot-2.7.1.jar:2.7.1]
ovsx-openvsx-server-1  |        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:734) ~[spring-boot-2.7.1.jar:2.7.1]
ovsx-openvsx-server-1  |        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:408) ~[spring-boot-2.7.1.jar:2.7.1]
ovsx-openvsx-server-1  |        at org.springframework.boot.SpringApplication.run(SpringApplication.java:308) ~[spring-boot-2.7.1.jar:2.7.1]
ovsx-openvsx-server-1  |        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) ~[spring-boot-2.7.1.jar:2.7.1]
ovsx-openvsx-server-1  |        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295) ~[spring-boot-2.7.1.jar:2.7.1]
ovsx-openvsx-server-1  |        at org.eclipse.openvsx.RegistryApplication.main(RegistryApplication.java:41) ~[classes/:na]
ovsx-openvsx-server-1  | Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'jobScheduler' defined in class path resource [org/jobrunr/spring/autoconfigure/JobRunrAutoConfiguration.class]: Unsatisfied dependency expressed through method 'jobScheduler' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'storageProvider' defined in class path resource [org/jobrunr/spring/autoconfigure/storage/JobRunrElasticSearchStorageAutoConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.jobrunr.storage.StorageProvider]: Factory method 'elasticSearchStorageProvider' threw exception; nested exception is ElasticsearchException[java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused]; nested: ExecutionException[java.net.ConnectException: Connection refused]; nested: ConnectException[Connection refused];

Do you have an idea of what could be wrong in the application.yml?

Thank you.

amvanbaren commented 1 year ago

In docker-compose.yml change image: elasticsearch:8.6.2 to image: elasticsearch:7.11.0

Looking at the exception, it seems like JobRunr is trying to use ElasticSearch to store its jobs, despite org.jobrunr.database.type: sql

amvanbaren commented 1 year ago

Maybe it is that the server can't connect to postgres?

aallrd commented 1 year ago

I actually forgot to mount the application.yml configuration file... 🙃

    volumes:
      - ./application.yml:/home/openvsx/server/config/application.yml

aallrd commented 1 year ago

Okay I am able to have something up and running with the below docker-compose.yml:

version: "3.9"
services:
  openvsx-webui:
    image: ghcr.io/eclipse/openvsx-webui:v0.9.5
    ports:
      - "3000:3000"
    depends_on:
      openvsx-server:
        condition: service_healthy
  openvsx-server:
    image: ghcr.io/eclipse/openvsx-server:v0.9.5
    ports:
      - "8080:8080"
    healthcheck:
      test: ["CMD-SHELL", "curl http://openvsx-server:8080"]
      interval: 10s
      timeout: 10s
      retries: 6
    volumes:
      - ./application.yml:/home/openvsx/server/config/application.yml
    depends_on:
      elasticsearch:
        condition: service_healthy
      postgres:
        condition: service_healthy
  elasticsearch:
    image: elasticsearch:8.6.2
    environment:
      - xpack.security.enabled=false
      - discovery.type=single-node
    ports:
      - "9200:9200"
      - "9300:9300"
    healthcheck:
      test: ["CMD-SHELL", "curl http://elasticsearch:9200/_cluster/health?wait_for_status=green"]
      timeout: 60s
  postgres:
    image: postgres:14.7-alpine
    environment:
      - POSTGRES_PASSWORD=postgres
    ports:
      - "5432:5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 10s
      retries: 6
networks:
  default:
    name: openvsx-network

As you mentioned, the Publish (http://localhost:3000/user-settings/extensions) and Login URLs do not work.

I am now trying to create a namespace/publish my package, using the super_token you mentioned before but it fails:

$ ovsx -p super_token -r http://localhost:8080/ create-namespace test
❌  Invalid access token.

Can you share more details about how to publish packages in this configuration ?

amvanbaren commented 1 year ago

I think you need a user and personal access token. This is from the dev configuration:

INSERT INTO user_data(id, login_name, role) VALUES(1001, 'super_user', 'admin');
INSERT INTO personal_access_token(id, active, description, value, user_data) VALUES(1002, TRUE, 'For publishing test extensions', 'super_token', 1001);

I think you don't need to set role: admin, because the account is just for publishing.

aallrd commented 1 year ago

Thank you for these DB commands, they will be useful to manually create other users/tokens. I was able to create a namespace and submit an extension, but I have a few questions:

Is there a way to create a multi-string namespace (eg: "ACME co") ? The create-namespace command complains about an invalid namespace:
```
$ ovsx -p super_token -r http://localhost:8080/ create-namespace "ACME co"
❌  Invalid namespace name: ACME co
```

One of our vsix package has a dependency on another extension published on the Microsoft Marketplace, the publish commands complains that it cannot resolve the dependency and therefore fails to upload. Is it expected?

$ ovsx -p super_token -r http://localhost:8080/ publish test.vsix
❌  Cannot resolve dependency: matepek.vscode-catch2-test-adapter
See the documentation for more information:
https://github.com/eclipse/openvsx/wiki/Publishing-Extensions

After publishing the vsix package, I cannot see it on the web portal. Trying to publish it a second time tells me it's inactive and therefore not visible, what should I do ?

$ ovsx -p super_token -r http://localhost:8080/ publish test.vsix
❌  Extension test is already published, but is currently inactive and therefore not visible.
See the documentation for more information:
https://github.com/eclipse/openvsx/wiki/Publishing-Extensions

amvanbaren commented 1 year ago

* Is there a way to create a multi-string namespace (eg:  "ACME co") ? The `create-namespace` command complains about an invalid namespace:

Yes, you can use -.

* One of our vsix package has a dependency on another extension published on the Microsoft Marketplace, the `publish` commands complains that it cannot resolve the dependency and therefore fails to upload. Is it expected?

Yes, that's expected. You need to publish the dependency first and then try again.

* After publishing the vsix package, I cannot see it on the web portal. Trying to publish it a second time tells me it's inactive and therefore not visible, what should I do ?

extension file resources are processed async. It can take a little while before it becomes available on the web portal

aallrd commented 1 year ago

Using a '-' will just create "ACME-co", it's not the same as "ACME co". I have published it more than one hour ago now and it's still not visible, is that expected? I am puzzled by the error message:

❌  Extension test is already published, but is currently inactive and therefore not visible.

What does inactive mean ?

amvanbaren commented 1 year ago

Using a '-' will just create "ACME-co", it's not the same as "ACME co".

That's true, but spaces are not allowed in the namespace name. The displayName can have spaces.

I have published it more than one hour ago now and it's still not visible, is that expected?

No, something went wrong. I'd clear the database and try again.

What does inactive mean ?

The extension can't be retrieved and doesn't show up in search results.

aallrd commented 1 year ago

I cleared the DB, published again and found this in the log:

ovsx-postgres-1        | 2023-03-27 15:32:07.886 UTC [82] ERROR:  insert or update on table "file_resource" violates foreign key constraint "file_resource_extension_fkey"
ovsx-postgres-1        | 2023-03-27 15:32:07.886 UTC [82] DETAIL:  Key (extension_id)=(4) is not present in table "extension_version".
ovsx-postgres-1        | 2023-03-27 15:32:07.886 UTC [82] STATEMENT:  insert into file_resource (content, extension_id, name, storage_type, type, id) values ($1, $2, $3, $4, $5, $6)
ovsx-openvsx-server-1  | 2023-03-27 15:32:07.888  WARN 1 --- [         task-1] o.h.engine.jdbc.spi.SqlExceptionHelper   : SQL Error: 0, SQLState: 23503
ovsx-openvsx-server-1  | 2023-03-27 15:32:07.888 ERROR 1 --- [         task-1] o.h.engine.jdbc.spi.SqlExceptionHelper   : ERROR: insert or update on table "file_resource" violates foreign key constraint "file_resource_extension_fkey"

Seems like indeed it failed to insert in the DB despite the success message:

$ ovsx -p super_token -r http://localhost:8080/ publish test-0.0.1.vsix
🚀  Published ACME.test v0.0.1

Do you have an idea what could be the issue?

Also, is there a way to delete things (namespaces, packages, ...) from the command line since the UI is not available?

amvanbaren commented 1 year ago

Do you have an idea what could be the issue?

That happens sometimes. This method is retryable, so it doesn't directly mean the publish operation failed.

Also, is there a way to delete things (namespaces, packages, ...) from the command line since the UI is not available?

No, there isn't. You can clear the database.

aallrd commented 1 year ago

Do you have a demo/reference VSIX package I could try to test with? I am completely stuck now, I can browse the DB manually but it does not tell me much. I can see an entry in the extension and _extensionversion tables, but nothing in _fileresource.

amvanbaren commented 1 year ago

https://open-vsx.org/api/robole/file-bunny/1.3.6/file/robole.file-bunny-1.3.6.vsix

aallrd commented 1 year ago

I am still not able to make it work using your example extension:

wget https://open-vsx.org/api/robole/file-bunny/1.3.6/file/robole.file-bunny-1.3.6.vsix
$ ovsx -p super_token -r http://localhost:8080/ create-namespace robole
🚀  Created namespace robole
$ ovsx -p super_token -r http://localhost:8080/ publish robole.file-bunny-1.3.6.vsix
🚀  Published robole.file-bunny v1.3.6
$ ovsx -p super_token -r http://localhost:8080/ publish robole.file-bunny-1.3.6.vsix
❌  Extension robole.file-bunny 1.3.6 is already published, but is currently inactive and therefore not visible.
See the documentation for more information:
https://github.com/eclipse/openvsx/wiki/Publishing-Extensions

I have the same exception that I mentioned before in the logs, and nothing is displayed in the UI. I can see an entry in the extension and _extensionversion tables, but nothing in _fileresource.

$ psql -h localhost -p 5432 -U postgres -c "SELECT * from extension;"
 id | average_rating | download_count |    name    | namespace_id |              public_id               | active |       published_date       |     last_updated_date      | review_count
----+----------------+----------------+------------+--------------+--------------------------------------+--------+----------------------------+----------------------------+--------------
  3 |                |              0 | file-bunny |            1 | 3c6d5a39-673e-4689-b418-6874017dedea | f      | 2023-03-28 08:34:24.476106 | 2023-03-28 08:34:24.476106 |
(1 row)
$ psql -h localhost -p 5432 -U postgres -c "SELECT * from file_resource;"
 id |     type     |

Can you please share what version of PostgreSQL and Elasticsearch you are using in production so that I align with these versions?

I have attached my docker-compose.yml, application.yml and ovsx.sql files for reproduction: ovsx-compose-resources.tar.gz

I don't know how to move forward on this issue.

amvanbaren commented 1 year ago

I get the same error messages when opening openvsx in a Gitpod workspace. I'll try to find the cause of the issue there.

In the meantime you can try PostgreSQL 10.17 and ElasticSearch 7.11.0.

aallrd commented 1 year ago

Using these versions of PostgreSQL and and ElasticSearch, I am able to see the extension on the portal 👍 .

I am still getting the aforementioned errors in the logs, so it does not seem to be related to this specific issue.

The portal weirdly displays "0 results" on the homepage (do you know why?), but searching for the extension yields results and I can browse the extension details.

amvanbaren commented 1 year ago

I am still getting the aforementioned errors in the logs, so it does not seem to be related to this specific issue.

Ok, thanks for the feedback.

The portal weirdly displays "0 results" on the homepage (do you know why?)

Search results are cached for 10 minutes. When you do CTRL+F5 it should show the added extension.

amvanbaren commented 1 year ago

I fixed the publish errors that I got on Gitpod. It might also fix the publishing error that you're seeing. You can try it yourself with this Docker image: docker.io/amvanbaren/openvsx-server:9d72a82

aallrd commented 1 year ago

I confirm that I am not seeing the aforementioned errors in the log using docker.io/amvanbaren/openvsx-server:9d72a82. Also, I think the issue I was experiencing comes from Elasticsearch since I am successfully using a postgres:14.7-alpine image instead of PostgreSQL 10.17 you recommended.

aallrd commented 1 year ago

As mentioned in #706, I am now trying to host this setup internally at https://openvsx.foo.com. Since both the server and the UI must be accessible from the same port with this deployment model, I built a custom openvsx-server image using the Dockerfile you mentioned to create a single image (server + UI). I only changed the way the UI code is fetched in the Dockerfile (clone instead of copy):

# Building and packaging the UI
ENV WEBSITE_VERSION 0.9.5
#COPY . /workdir
RUN git clone --branch v${WEBSITE_VERSION} --depth 1 https://github.com/eclipse/openvsx.git /workdir

RUN /usr/bin/yarn --cwd webui \
  && /usr/bin/yarn --cwd webui build

# Main image derived from openvsx-server
FROM ghcr.io/eclipse/openvsx-server:v0.9.5

COPY --from=builder --chown=openvsx:openvsx /workdir/webui/static/ BOOT-INF/classes/static/
#COPY --from=builder --chown=openvsx:openvsx /workdir/configuration/ config/

I updated my _dockercompose.yml with this new custom image (removed the UI service) and configured my reverse proxy to listen on http://localhost:8080/ but I am unable to access the UI (blank page).

In the web browser console when opening https://openvsx.foo.com/ I have the below error:

The resource from “https://openvsx.foo.com/bundle.js” was blocked due to MIME type (“application/json”) mismatch (X-Content-Type-Options: nosniff).

I cannot find the file bundle.js in my custom built openvsx-server image, what did I miss?

amvanbaren commented 1 year ago

RUN git clone --branch v${WEBSITE_VERSION} --depth 1 https://github.com/eclipse/openvsx.git /workdir

You need to do ./workdir (notice the dot) to indicate the current directory.

aallrd commented 1 year ago

It does clone in /workdir:

Step 8/12 : RUN git clone --branch v${WEBSITE_VERSION} --depth 1 https://github.com/eclipse/openvsx.git /workdir
 ---> Running in 334ffaf15f7f
Cloning into '/workdir'...
Note: switching to '1521526981133b5d50bd8729c8ce2b09ab2a6210'.
[...]
Removing intermediate container 334ffaf15f7f
 ---> af21a2fd93c3
Step 9/12 : RUN ls -ltr /workdir
 ---> Running in 655a538aca84
total 44
drwxr-xr-x 6 root root  4096 Mar 31 11:49 webui
drwxr-xr-x 5 root root  4096 Mar 31 11:49 server
drwxr-xr-x 2 root root  4096 Mar 31 11:49 doc
drwxr-xr-x 5 root root  4096 Mar 31 11:49 cli
-rw-r--r-- 1 root root  8042 Mar 31 11:49 README.md
-rw-r--r-- 1 root root 14198 Mar 31 11:49 LICENSE
-rw-r--r-- 1 root root  1708 Mar 31 11:49 CONTRIBUTING.md
Removing intermediate container 655a538aca84

With your ./workdir suggestion the yarn build fails:

Step 10/12 : RUN /usr/bin/yarn --cwd webui   && /usr/bin/yarn --cwd webui build
 ---> Running in 43ec2b86f8b8
yarn install v1.22.19
error Directory "/workdir/webui" doesn't exist

aallrd commented 1 year ago

The file bundle.js seems to be available after the yarn build:

Step 10/13 : RUN /usr/bin/yarn --cwd webui   && /usr/bin/yarn --cwd webui build
 ---> Using cache
 ---> 7ac206f87cbe
Step 11/13 : RUN find . -name bundle.js
 ---> Running in 9fccc2902240
./webui/node_modules/ajv/scripts/bundle.js

But only the static folder is copied at the end of the Dockerfile, is that expected?

COPY --from=builder --chown=openvsx:openvsx /workdir/webui/static/ BOOT-INF/classes/static/

amvanbaren commented 1 year ago

But only the static folder is copied at the end of the Dockerfile, is that expected?

Yes, but for the webui to generate bundle.js you need to add && yarn --cwd webui build:default , so:

RUN /usr/bin/yarn --cwd webui \
  && /usr/bin/yarn --cwd webui build \
  && /usr/bin/yarn --cwd webui build:default

aallrd commented 1 year ago

Okay that did the trick even though I don't understand why it has to be different from the project's Dockerfile. It fixed the blank UI issue, I am now able to access the service at https://openvsx.foo.com/. I successfully pushed the demo extension and my internal test extension to the registry.

ovsx -p super_token -r https://openvsx.foo.com publish test-0.0.1.vsix
🚀  Published ACME.test v0.0.1

Now that I moved to an isolated network without internet access I am facing a new issue: extensions information is not loading when I browse them. I expect it to be broken for external extensions that reference outgoing links, but not for internal extensions referencing internal URLs,

I am observing two behaviors:

Browsing an external extension (robole.file-bunny): blank page without any details, just the purple loading bar.
Browsing an internal extension (ACME.test): Details about the name, publisher, license, description, rating, .... but all the tabs overview/changes/ratings are empty and loading indefinitely.

Do you have an idea what could be the issue?

amvanbaren commented 1 year ago

Can you open the network tab in the browser developer tools and search for README.md?

aallrd commented 1 year ago

From the web console I get:

Blocked loading mixed active content “http://openvsx.foo.com/api/ACME/test/0.0.1/file/README.md”

amvanbaren commented 1 year ago

~~Is the mime-type application/octet-stream?~~

aallrd commented 1 year ago

I am not sure how to check that, where can I see it? But the issue seems to be that the URL is fetched with the http protocol instead of https (like the origin).

amvanbaren commented 1 year ago

I am not sure how to check that, where can I see it?

Nevermind, that was my first guess.

amvanbaren commented 1 year ago

You can set https as value for the X-Forwarded-Proto request header in the reverse proxy, so that the Open VSX server generates the right URLs.

aallrd commented 1 year ago

I added the proxy_set_header X-Forwarded-Proto $scheme; line to my Nginx configuration. I restarted the openvsx server, I am now getting another error in the web console:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://openvsx.foo.com:80/api/ACME/test/0.0.1/file/README.md. (Reason: CORS request did not succeed). Status code: (null).

Shouldn't the port be 443 there ?

amvanbaren commented 1 year ago

You also need to set X-Forwarded-Host to openvsx.foo.com:443

aallrd commented 1 year ago

That did the trick. 👍 I am now able to browse the extension with all details, both internal/external ones. I was also able to install the extensions after configuring the gallery in VSCodium. Thank you for the support, I'll try to document all these details next week and how I setup this instance.

amvanbaren commented 1 year ago

I'll try to document all these details next week and how I setup this instance.

That would be much appreciated. Looking forward to it!

oloedpaper commented 1 year ago

Here is the complete Dockerfile

# Builder image to compile the website
FROM ubuntu as builder

WORKDIR /workdir

RUN apt-get update \
  && apt-get install --no-install-recommends -y \
    bash \
    ca-certificates \
    git \
    curl \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

# See https://github.com/nodesource/distributions/blob/main/README.md#debinstall
RUN curl -sSL https://deb.nodesource.com/setup_14.x | bash - \
  && apt-get install -y nodejs

RUN npm install --global yarn@1.*

# bump to update website
ENV WEBSITE_VERSION 0.9.5
#COPY . /workdir
RUN git clone --branch v${WEBSITE_VERSION} --depth 1 https://github.com/eclipse/openvsx.git /workdir
RUN /usr/bin/yarn --cwd webui \
  && /usr/bin/yarn --cwd webui build \
  && /usr/bin/yarn --cwd webui build:default

# Main image derived from openvsx-server
FROM ghcr.io/eclipse/openvsx-server:2252936

COPY --from=builder --chown=openvsx:openvsx /workdir/webui/static/ BOOT-INF/classes/static/
#COPY --from=builder --chown=openvsx:openvsx /workdir/configuration/ config/

amvanbaren commented 1 year ago

Thanks @oloedpaper. I've added a slightly modified version of the DockerFile to the wiki: https://github.com/eclipse/openvsx/wiki/Deploying-Open-VSX#getting-started

makar723 commented 9 months ago

Hello guys! Is there any doc, how can I deploy OpenVSX behind the nginx? I have tried to use traefik with wildcard certificate and nginx with the same domain, but in both cases I getting the same error (via https): Error An unexpected error occurred: Unexpected token '<', "<!DOCTYPE "... is not valid JSON I've tried to build docker image and also tried to pull image v0.14.1. When i use http and port 3000 I don't see such error and could publish test app via ovsx. Maybe you have complete guide for the deployment? We are deploying the application in the cloud. Thank you in advance

eclipse / openvsx

Setting up a private Open VSX registry #703