Support storing pulumi state in terraform http state compatible backends

[Place1]

Gitlab 13 has added a cool new feature that provides direct support for Terraform HTTP State. This new feature means that terraform users can use Gitlab as their state backend with some nice integration with Gitlab CI and the general project/repo stuff in Gitlab.

https://about.gitlab.com/releases/2020/05/22/gitlab-13-0-released/#gitlab-http-terraform-state-backend

I'm interested to scope out support for the Terraform HTTP State provider in pulumi. It would be pretty cool to run a pulumi login rest://myserver.com/state and be up and running with pulumi.

This is probably a useful feature for teams working at organisations that already have their own custom terraform HTTP state backends; they could potentially start using pulumi and integrate with this existing state solution.

The usage docs from terraform are here: https://www.terraform.io/docs/backends/types/http.html

Here's an example http state implementation: https://github.com/mikalstill/junkcode/tree/master/terraform/remote_state

If this is something that pulumi would accept a contribution for i'd be happy to put together a working draft, demo and PR to kick this off.

[Place1]

I believe i'll be able to implement this fairly quickly on top of the current filestate backend as another gocloud blob driver.

[leezen]

Awesome -- thanks for the update!

[ryan4yin]

awesome!

[logileifs]

Is there any news on this ?

[jay13jay]

is anyone still working on this? I am trying to make the case to use pulumi on some of my projects but being able to use gitlab as a backend state manager would make my attempts a lot more successful...

[alexander-blackwell]

I would also be interested in utilizing Pulumi but would like to see backend state management support in Gitlab first.

[hbjydev]

This would be really cool to have, because I really wanna switch from Terraform to Pulumi, but we need our state to be managed by us, so being able to use GitLab’s TF state provider to make this work would be fantastic :)

any updates on it?

[mattheews]

I'm Also looking forward to enable this feature! :muscle:

[rbcb-bedag]

We are also interested in this feature. Any updates on this ?

[yellowhat]

My workaround is to store the state as a gitlab package:

• .gitlab-ci.yml

stages:
  - infra

infra:
  stage: build
  image: docker.io/python:3.12.3-slim
  variables:
    PULUMI_CONFIG_PASSPHRASE: foo  # Should be an env var
    PULUMI_STATE_DIR: "${CI_PROJECT_DIR}/state"
    PULUMI_STACK: me
  before_script:
    - ./scripts/install.sh
    - ./scripts/pulumi-state.sh download
    - ./scripts/pulumi-setup.sh
  script:
    - pulumi preview
    - pulumi up --yes
    - pulumi stack output --show-secrets
  after_script:
    - ./scripts/pulumi-state.sh upload

• scripts/pulumi-setup.sh:

#!/usr/bin/env bash
set -euo pipefail

echo "[INFO] Pulumi - login"
mkdir -p "$PULUMI_STATE_DIR"
pulumi login "file://${PULUMI_STATE_DIR}"
echo ""

echo "[INFO] Pulumi - create $PULUMI_STACK stack"
pulumi stack select --create "$PULUMI_STACK"
echo ""

echo "[INFO] Pulumi - stacks:"
pulumi stack ls
echo ""

• scripts/pulumi-state.sh:

#!/usr/bin/env bash
# Download/Upload pulumi state
set -xeuo pipefail

PACKAGE_URL="${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/generic/state/0.0.0/state.tar.gz"
PACKAGE_TAR="${PACKAGE_URL##*/}"

case "$1" in
    "download")
        echo "[INFO] Check if exist"
        curl \
            --header "JOB-TOKEN: $CI_JOB_TOKEN" \
            --output /dev/null \
            --silent \
            --head \
            --fail \
            "$PACKAGE_URL" || exit 0
        echo "[INFO] Download state"
        curl \
            --header "JOB-TOKEN: $CI_JOB_TOKEN" \
            "$PACKAGE_URL" | tar xz -C /
        ;;
    "upload")
        echo "[INFO] Compressing state"
        tar czf "$PACKAGE_TAR" -C "$PULUMI_STATE_DIR" "$PULUMI_STATE_DIR/"
        echo "[INFO] Upload state"
        curl \
            --upload-file "$PACKAGE_TAR" \
            --header "JOB-TOKEN: $CI_JOB_TOKEN" \
            "$PACKAGE_URL"
        ;;
    *)
        echo "Invalid option: $1"
        exit 1
        ;;
esac