Informatievlaanderen / VSDS-LDES-E2E-testing

End-to-end tests and tooling needed for testing LDES components build as part of VSDS
European Union Public License 1.2
5 stars 2 forks source link

ldes server ingest members fail #99

Closed zargha closed 1 year ago

zargha commented 1 year ago

The e2e test https://github.com/Informatievlaanderen/VSDS-LDES-E2E-testing/tree/main/e2e-test/use-cases/gipod/2.ingest-ldes fails. The members are not received in mongodb and in ldes server.

docker-compose.yaml:

version: '3.3'
services:
  ldes-server-simulator:
    container_name: ${USECASE_NAME:-simulator-workflow-server-mongo}_ldes-server-simulator
    image: ghcr.io/informatievlaanderen/ldes-server-simulator:${LDES_SERVER_SIMULATOR_TAG:-20221012T1428}
    environment:
      - BASEURL=http://ldes-server-simulator
      - SEED=/tmp/data
    ports:
      - ${LDES_SERVER_SIMULATOR_PORT:-9011}:80
    volumes:
      - type: bind
        source: ${LDES_SERVER_SIMULATOR_SEED_FOLDER}
        target: /tmp/data
    networks:
      - ldes
  nifi-workflow:
    image: ghcr.io/informatievlaanderen/ldes-workbench-nifi:${LDES_WORKBENCH_NIFI_TAG:-20221024T080100}
    container_name: ${USECASE_NAME:-simulator-workflow-server-mongo}_ldes-workbench-nifi
    environment:
      - SINGLE_USER_CREDENTIALS_USERNAME=${SINGLE_USER_CREDENTIALS_USERNAME}
      - SINGLE_USER_CREDENTIALS_PASSWORD=${SINGLE_USER_CREDENTIALS_PASSWORD}
    ports:
      - ${NIFI_UI_PORT:-8443}:8443
    networks:
      - ldes
  ldes-server:
    container_name: ${USECASE_NAME:-simulator-workflow-server-mongo}_ldes-server
    image: ghcr.io/informatievlaanderen/ldes-server:${LDES_SERVER_TAG:-20221021T1210}
    environment:
      - SPRING_DATA_MONGODB_DATABASE=${SPRING_DATA_MONGODB_DATABASE}
      - LDES_COLLECTIONNAME=${LDES_COLLECTIONNAME}
      - LDES_MEMBERTYPE=${LDES_MEMBERTYPE}
      - SPRING_DATA_MONGODB_HOST=ldes-mongodb
      - SPRING_DATA_MONGODB_PORT=27017
      - LDES_HOSTNAME=http://localhost:${LDES_SERVER_PORT:-8080}
      - LDES_SHAPE=${LDES_SHAPE:-}
      - VIEW_TIMESTAMPPATH=${VIEW_TIMESTAMPPATH:-http://www.w3.org/ns/prov#generatedAtTime}
      - VIEW_VERSIONOFPATH=${VIEW_VERSIONOFPATH:-http://purl.org/dc/terms/isVersionOf}

      # default view by time
      - VIEWS_0_NAME=${VIEWS_0_NAME:-$LDES_COLLECTIONNAME-by-time}
      - VIEWS_0_FRAGMENTATIONS_0_NAME=timebased
      - VIEWS_0_FRAGMENTATIONS_0_CONFIG_MEMBERLIMIT=${VIEWS_0_FRAGMENTATIONS_0_CONFIG_MEMBERLIMIT:-100}

      # # Alternative view by location and time
      # - VIEWS_0_NAME=${VIEWS_0_NAME:-$LDES_COLLECTIONNAME-by-location-and-time}
      # - VIEWS_0_FRAGMENTATIONS_0_NAME=geospatial
      # - VIEWS_0_FRAGMENTATIONS_0_CONFIG_PROJECTION=lambert72
      # - VIEWS_0_FRAGMENTATIONS_0_CONFIG_MAXZOOMLEVEL=${VIEWS_0_FRAGMENTATIONS_0_CONFIG_MAXZOOMLEVEL:-15}
      # - VIEWS_0_FRAGMENTATIONS_0_CONFIG_FRAGMENTERPROPERTY="http://www.opengis.net/ont/geosparql#asWKT"
      # - VIEWS_0_FRAGMENTATIONS_1_NAME=timebased
      # - VIEWS_0_FRAGMENTATIONS_1_CONFIG_MEMBERLIMIT=${VIEWS_0_FRAGMENTATIONS_1_CONFIG_MEMBERLIMIT:-50}

      # # # Additional view by time
      # # - VIEWS_1_NAME=${VIEWS_1_NAME:-$LDES_COLLECTIONNAME-by-time}
      # # - VIEWS_1_FRAGMENTATIONS_0_NAME=timebased
      # # - VIEWS_1_FRAGMENTATIONS_0_CONFIG_MEMBERLIMIT=${VIEWS_1_FRAGMENTATIONS_0_CONFIG_MEMBERLIMIT:-100}
    ports:
      - ${LDES_SERVER_PORT:-8080}:8080
    networks:
      - ldes
    depends_on:
      - ldes-mongodb
  ldes-mongodb:
    container_name: ${USECASE_NAME:-simulator-workflow-server-mongo}_ldes-mongodb
    image: mongo:${MONGODB_TAG:-5.0.13}
    ports:
      - ${MONGODB_PORT:-27017}:27017
    volumes:
      - type: bind
        source: ${MONGODB_DATA_FOLDER}
        target: /data/db
    networks:
      - ldes
networks:
  ldes:
    name: ${USECASE_NAME:-simulator-workflow-server-mongo}_network

env.user:

# Use case
USECASE_NAME=gipod-ingest-ldes

# Simulator compose env vars
LDES_SERVER_SIMULATOR_SEED_FOLDER=../../../../ldes-server-simulator/data/gipod

# Workflow compose env vars
SINGLE_USER_CREDENTIALS_USERNAME=
SINGLE_USER_CREDENTIALS_PASSWORD=

# Server compose env vars
SPRING_DATA_MONGODB_DATABASE=gipod
LDES_COLLECTIONNAME=mobility-hindrances
LDES_MEMBERTYPE="https://data.vlaanderen.be/ns/mobiliteit#Mobiliteitshinder"
LDES_SHAPE=https://private-api.gipod.test-vlaanderen.be/api/v1/ldes/mobility-hindrances/shape

# NOTE: by default, there is one view available with timebased fragmentation
# to change this, please uncomment the relevant section(s) in the docker-compose.yml file
# currently, you can use an alternative geospatial fragmentation and additionally a second, timebased view
VIEWS_0_FRAGMENTATIONS_0_CONFIG_MEMBERLIMIT=1500

# Mongo compose env vars
MONGODB_DATA_FOLDER=data/gipod/replicate
# e.g. ~/data/gipod/replicate
rorlic commented 1 year ago

Yesterday, I verified and fixed all E2E tests, including this one. Please retry (with latest tags).

zargha commented 1 year ago

Yesterday, I verified and fixed all E2E tests, including this one. Please retry (with latest tags).

Verified with latest tag. issue is solved.