opensanctions / yente

API for OpenSanctions with support for entity search and bulk matching of data collections. Supports Reconciliation API spec.
https://www.opensanctions.org/docs/yente/
MIT License
71 stars 29 forks source link

Added Support for Yente with Elasticsearch on CapRover: A Template for Easy Deployment #408

Open pescheck-devops opened 8 months ago

pescheck-devops commented 8 months ago

We are unsure where this should be directed, but we aim to integrate Yente and the indexing service with Elasticsearch within CapRover for experimentation. We have developed a template.

How to install this on CapRover?

  1. Navigate to Apps
  2. Click on "One Click Apps/Databases"
  3. Navigate to the very bottom of the list, and click on the last item, called >> TEMPLATE <<
  4. Copy the following section to the box:
captainVersion: 4
caproverOneClickApp:
    instructions:
        start: Starting of Yente.
        end: Yente is deployed.
    variables:
        - id: $$cap_elasticsearch_version
          label: 'Elasticsearch Version Tag'
          description: 'Check out the releases overview: https://hub.docker.com/_/elasticsearch'
          defaultValue: 8.4.1
          validRegex: /^([^\s^\/])+$/
        - id: $$cap_elasticsearch_cluster_name
          label: Cluster Name
          description: Only nodes within the same cluster name can be combined
          defaultValue: elasticsearch-cluster
          validRegex: /^([^\s^\/])+$/
        - id: $$cap_elasticsearch_discovery_type
          label: Discovery Type
          description: Discovery type, for a single node cluster use `single-node`, otherwise `multi-node`
          defaultValue: single-node
          validRegex: /^([^\s^\/])+$/
        - id: $$cap_elasticsearch_security_enabled
          label: Security Enabled
          defaultValue: 'false'
          description: 'When you enable this option, Elasticsearch will create a random password (see startup logs) for the `elastic` user and create SSL certificates required for authentication. It is recommended to leave this off for a quick setup. Warning: make sure to enable HTTP Basic Auth in CapRover!'
          validRegex: /^([^\s^\/])+$/
        - id: $$cap_container_index_port
          label: Container TCP Port
          defaultValue: '9200'
          description: Internal port for Elasticsearch the container listens to.
          validRegex: /^([0-9])+$/
        - id: $$cap_container_app_port
          label: Container TCP Port
          defaultValue: '8000'
          description: Internal port for Yente the container listens to.
          validRegex: /^([0-9])+$/
    displayName: Yente
    isOfficial: true
    description: Yente is an open source data match-making API. The service provides several HTTP endpoints to search, retrieve or match FollowTheMoney entities, including people, companies or vessels that are subject to international sanctions.
    documentation: Taken from https://github.com/opensanctions/yente

services:
  $$cap_appname-index:
    image: docker.elastic.co/elasticsearch/elasticsearch:$$cap_elasticsearch_version
    caproverExtra:
      notExposeAsWebApp: 'true'
      containerHttpPort: $$cap_container_index_port
    volumes:
      - $$cap_appname-index-elasticsearch-data:/usr/share/elasticsearch/data
    restart: always
    environment:
      CLI_JAVA_OPTS: -Xms512m -Xmx512m
      cluster.name: $$cap_elasticsearch_cluster_name
      discovery.type: $$cap_elasticsearch_discovery_type
      http.port: $$cap_container_index_port
      node.name: $$cap_appname-index
      xpack.security.enabled: $$cap_elasticsearch_security_enabled
  $$cap_appname-app:
    image: ghcr.io/opensanctions/yente:latest
    depends_on:
      - $$cap_appname-index
    environment:
      YENTE_ELASTICSEARCH_URL: http://srv-captain--$$cap_appname-index:9200
      YENTE_STATEMENT_API: "false"
      YENTE_UPDATE_TOKEN: ""
      YENTE_ELASTICSEARCH_INDEX: "$$cap_appname-index"
    healthcheck:
      test: [ "CMD", "curl", "-f", "http://localhost:8000/healthz" ]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 3s
    deploy:
      mode: replicated
      replicas: 1
      restart_policy:
        condition: on-failure
        delay: 10s
        max_attempts: 3
        window: 120s
    caproverExtra:
      containerHttpPort: $$cap_container_app_port
volumes:
  index-os-data: null

Additionally, we plan to submit a GitHub MR on repository https://github.com/caprover/one-click-apps/ to enable the CapRover community to explore this setup as well.

pudo commented 8 months ago

Heya! Thanks very much for suggesting the CapRover support configuration. I hadn't heard of the tool before, but it looks fascinating. To be very honest I am a bit worried about adding the support to our repo if our team doesn't have access to a running instance of the system (and capacity to try this out regularly). Does the integration into one-click-apps require the config to exist in our repo? Otherwise, perhaps we can just link out to that location?

pescheck-devops commented 8 months ago

Heya! Thanks very much for suggesting the CapRover support configuration. I hadn't heard of the tool before, but it looks fascinating. To be very honest I am a bit worried about adding the support to our repo if our team doesn't have access to a running instance of the system (and capacity to try this out regularly). Does the integration into one-click-apps require the config to exist in our repo? Otherwise, perhaps we can just link out to that location?

Heya @pudo,

Thanks for your feedback! Just wanted to give you a heads up about our efforts integrating Yente with CapRover. We're currently testing it thoroughly on our end and happy to report it actually works very nicely. Once we're confident it works flawlessly, we'll proceed with a MR on the CapRover one-click apps repo, ensuring the integration doesn’t require any direct support from your team.

Feel free to close this ticket if you prefer; we just wanted to keep you in the loop. We'll handle everything from testing to submission and let you know once it's all set for a public link.

Appreciate your understanding and looking forward to contributing to the community!

~ All the best

pudo commented 8 months ago

That's just amazing, thank you so much for contributing in this way! And please keep me in the loop - will definitely keep this ticket open. We just need to make sure we define the scope of what we support a bit, in order to not mis-direct the people who depend on our team for support :)