Using InfluxDB instead of CrateDB

[AngeloIapichino]

Hi guys,

I'm working in a project where some data are acquired through fiware environment storing them on CrateDB and some data are acquired through Apache Kafka storing them on InfluxDB.

Could be fine have only one database and we are evaluating if the fiware environment can store data in influxDB and if Apache Kafka can store data in CrateDB.

I'm working on the fiware side and i trying to use InfluxDB. I set the environment through docker-compose file:

version: '3'

services:

  orion:
    image: fiware/orion:${ORION_VERSION:-2.2.0}
    ports:
      - "1026:1026"
    command: -logLevel DEBUG -noCache -dbhost mongo -corsOrigin __ALL -notificationMode threadpool -statCounters -statSemWait -statTiming -statNotifQueue
    depends_on:
      - mongo
    healthcheck:
      test: ["CMD", "curl", "-f", "http://0.0.0.0:1026/version"]
      interval: 1m
      timeout: 10s
      retries: 3

  mongo:
    image: mongo:3.2.19
    ports:
      - "27017:27017"
    volumes:
      - mongodata:/data/db

  quantumleap:
    image: ${QL_IMAGE:-smartsdk/quantumleap}
    ports:
      - "8668:8668"
    depends_on:
      - mongo
      - orion
      - influxdb
    environment:
      - INFLUX_HOST=${INFLUX_HOST:-influxdb}
      - USE_GEOCODING=True
      - REDIS_HOST=redis
      - REDIS_PORT=6379

  influxdb:
    image: influxdb:latest
    ports:
      - "8083:8083"
      - "8086:8086"
      - "8090:8090"
    volumes:
      - influxdata:/data

  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    environment:
      - GF_INSTALL_PLUGINS=grafana-clock-panel,grafana-worldmap-panel,flant-statusmap-panel
      - GF_PANELS_DISABLE_SANITIZE_HTML=true
    depends_on:
      - influxdb
    volumes:
      - grafanadata:/var/lib/grafana

  redis:
    image: redis
    deploy:
      # Scaling Redis requires some extra work.
      # See https://get-reddie.com/blog/redis4-cluster-docker-compose/
      replicas: 1
    ports:
      - "6379:6379"
    volumes:
      - redisdata:/data

volumes:
  mongodata:
  influxdata:
  redisdata:
  grafanadata:

networks:
    default:
        driver_opts:
            com.docker.network.driver.mtu: ${DOCKER_MTU:-1400}

When i do some subscriptions or send data, the quantumleap logs give me these type of errors:

crate.client.exceptions.ConnectionError: No more Servers available, exception from last server: HTTPConnectionPool(host='crate', port=4200): Max retries exceeded with url: /_sql?error_trace=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f651bdff400>: Failed to establish a new connection: [Errno -2] Name does not resolve',))
172.23.0.5 - - [11/Oct/2019 12:43:29] "POST /v2/notify HTTP/1.1" 500 -
INFO:werkzeug:172.23.0.5 - - [11/Oct/2019 12:43:29] "POST /v2/notify HTTP/1.1" 500 -

It seems that Quantumleap search however a crate database. Where i'm wrong in the configuration?

Thank you in advance

[chicco785]

As by documentation, only crate and timescale (but only for injecting data currently) are currently supported.

[chicco785]

here comes, the long reply. influxbd was initially explored, but not considered because:

• open version did not provide ha
• there was not support for geojson queries

I doubt that this changed meanwhile, so Martel won't support it. of course, if you want to code the support, you are welcome to do so.

there is a factory that allows to add more backends.

there is also a bit of code you may reuse that was initially used for testing influx, but was never updated, so you will have to look into current translator implementations and change it. you can find the code here