influxdata / influxdb

Scalable datastore for metrics, events, and real-time analytics
https://influxdata.com
Apache License 2.0
28.94k stars 3.55k forks source link

[2.0] Release full-featured official Docker containers on DockerHub #16649

Closed randycoulman closed 3 years ago

randycoulman commented 4 years ago

Proposal:

Can we get official 2.0 beta Docker images released to DockerHub?

Current behavior:

We currently release official Docker images on DockerHub for 1.x (https://hub.docker.com/_/influxdb). In addition to running influxdb, these images also allow for automatic creation of a database and a number of users. There are no such images available for 2.0 beta.

Desired behavior:

Have similar images for 2.0 beta as we do for 1.x, including the ability to easily provision an org, bucket, and user. In addition, because 2.0 requires a token for API requests, we'll need to come up with a secure way to make that token available for subsequent API calls after provisioning.

Alternatives considered:

We have "bare bones" images available at https://quay.io/repository/influxdb/influxdb?tag=latest&tab=tags, but they require additional work to provision. In addition, because 2.0 requires a token for making API calls, we need to come up with a solution for acquiring that token and making it available for subsequent API calls in a secure way.

There is v2 API endpoint for doing the provisioning at /api/v2/setup (https://v2.docs.influxdata.com/v2.0/api/#operation/PostSetup) and it returns a JSON payload containing the token. Extracting this token from some kind of command-line utility requires additional tools like jq.

Use case:

These images would be generally useful to allow people to get an instance of influxdb up and running quickly.

The particular use case I'm looking at right now is adding v2 support to the telemetry_influxdb project. This library is currently tested by spinning up a v1 Docker container using environment variables to provision a known DB and user/password. The tests then use those known values to run the library (which uses the /write API) and verify that it wrote the data correctly (using /query). See https://github.com/ludwikbukowski/telemetry_influxdb/blob/master/Makefile for how it runs the container.

To test v2 similarly, we'd need a way to spin up a v2 container, provision it, and get access to the token so that we can run the library and verify its operations. We're currently using an image from quay.io as above, and then doing a bunch of extra work to provision, store the token, and make it available to the tests.

This issue is somewhat related to #15761.

drdelambre commented 4 years ago

here's the provisioning script we use internally for development, maybe we can add it to an entrypoint script?

#! /bin/bash

echo "Waiting for the api to spin up"
while [[ "$(curl -s --insecure -o /dev/null -w ''%{http_code}'' http://influxdb:9999)" != "200" ]]; do sleep 5; done

echo "Provisioning from environment variables"
curl -X POST -H "Content-type: application/json" \
  -o ./auth-result.json \
  -w "%{http_code}\n" \
  -d "{\"username\":\"${INFLUX_USER}\",\"password\":\"${INFLUX_PASS}\",\"org\":\"${INFLUX_ORG}\",\"bucket\":\"${INFLUX_BUCKET}\"}" \
  http://influxdb:9999/api/v2/setup 2>/dev/null >> ./status_code

read -r CODE < ./status_code
read -r BODY < ./auth-result.json

if [[ $CODE =~ ^20 ]]
then
  TOKEN="$(jq -r .auth.token <<< $BODY)"

  echo "export INFLUX_TOKEN=${TOKEN}" > /conf/influx-token.env
else
  echo "already provisioned"
fi
jsternberg commented 4 years ago

Probably the biggest requirement for this is that we need to gpg sign the packages. The official docker images need to be downloaded and verified with gpg or some other method of artifact verification (but gpg is the easiest for that).

At the moment, we sign the 1.x artifacts using adhoc methods that depend on the product from a random server that has the gpg key. InfluxDB 1.x is the only one that auto-signs itself. If we can add the gpg signing to the release process, that would make the official docker images pretty trivial to do.

Maybe we can open up a different issue for gpg signing the 2.0 artifacts if they aren't already signed?

bozic commented 4 years ago

Hello, Is there any progress on this? We would like to rebuild our own Docker image for v2 and remove Chronograf for V2 since we do not use it, so it would be very helpfull if the source of official build for V2 is available somewhere...

skapin commented 4 years ago

@bozic

this is my working draft

docker compose

version: '3'
services:
  # Define a Telegraf service
  telegraf:
    networks:
      - internal
    labels:
      - traefik.enable=false
    image: telegraf:latest
    volumes:
      - ./tick/telegraf.conf:/etc/telegraf/telegraf.conf:ro
    restart: always
    hostname: 'telegraf-boucle-logistique'
    links:
      - influxdb
    ports:
      - "8092:8092/udp"
      - "8094:8094"
      - "8125:8125/udp"
    env_file: .env-tick
  # Define an InfluxDB service
  influxdb:
    networks:
      - internal
    env_file: .env-tick
    labels:
      - traefik.enable=false
    restart: always
    # image: influxdb:latest
    image: quay.io/influxdb/influxdb:2.0.0-beta
    volumes:
      - influxdata:/var/lib/influxdb
      - ./influxdata:/root/.influxdbv2
      # - ./tick/influxdb.conf:/etc/influxdb/influxdb.conf:ro
      # - ./tick/influx_init.iql:/docker-entrypoint-initdb.d/influx_init.iql
    ports:
      # - "8086:8086"
      - "9999:9999"
  influxdb-cli:
    image: quay.io/influxdb/influxdb:2.0.0-beta
    # image: influxdb:latest
    entrypoint:
      - influx
      - -host
      - influxdb
    links:
      - influxdb
    networks:
      - internal
    labels:
      - traefik.enable=false
    env_file: .env-tick
volumes:
  influxdata:
networks:
  proxy:
    external: true
  internal:
    external: false
Edo78 commented 3 years ago

Influxdb 2 is now official but I still can't find an official docker image even if here is stated

In the future, the latest tag for this image will point to the latest released implementation of influxdb 2.0. This will happen when the first general available release for 2.0 exists.

timhallinflux commented 3 years ago

@Edo78 -- that is correct. We are working through the upgrade process for Docker users to ensure that folks who pull the new image can upgrade successfully. In the meantime, we are publicizing the image we are hosting on quay.io. Instructions are here: https://docs.influxdata.com/influxdb/v2.0/get-started/ click on the Docker tab for more details.

Are you starting from scratch or are you upgrading?

Edo78 commented 3 years ago

I just started less then 48 hours ago so I can drop everything and start with version 2.0 Thank you for pointing out the instructions

timhallinflux commented 3 years ago

for others who might stumble upon this issue. Initial Docker image for new folks is here: quay.io/influxdb/influxdb:v2.0.1

We are working towards an image that will better support the upgrade process and the plan is to post that to Docker Hub. Stay tuned.

loganmc10 commented 3 years ago

Is there a way using the Quay images to auto provision a database? The 1.8 images had environment variables like INFLUXDB_DB that could be used to provision something on start

w4tsn commented 3 years ago

@timhallinflux is your development and design process for the new image / upgrade procedure publicly accessible?

agolovenko commented 3 years ago

@drdelambre Since auth is mandatory in Influxdb2, we need a way to run docker instance that is pre-initialized from environment variables. Otherwise, it's gonna be challenging to use this from CI like we used to with 1x. Some ideas for env vars:

Pierre-Lannoy commented 3 years ago

Hello all! Is there some news on this? Thanks.

alsargent commented 3 years ago

While we're working on this issue, can we update https://hub.docker.com/_/influxdb to indicate that we have builds on Quay.io?

It could literally be a bullet added under Quick reference along the lines of:

I'm asking because we recently had a user spend 2 hours trying to debug an issue because they thought v2 was on DockerHub, and didn't try to pull it from Quay.io.

alsargent commented 3 years ago

I talked to Tim about this. We can hold off on my request above (to update our quay.io page) since we should have a solution for DockerHub very soon.

tilchl commented 3 years ago

@drdelambre Since auth is mandatory in Influxdb2, we need a way to run docker instance that is pre-initialized from environment variables. Otherwise, it's gonna be challenging to use this from CI like we used to with 1x. Some ideas for env vars:

  • token
  • org
  • bucket

Is this going to be implemented?

danxmoran commented 3 years ago

@tilchl yes, it is WIP now

danxmoran commented 3 years ago

Will be closed by docker-library/official-images#9594