error: Earthfile line 401:4 copy artifact: apply build +gtfs-get-mobilitydb: earthfile2llb for +gtfs-get-mobilitydb: parse target name +debian:bookworm-slim: target name debian:bookworm-slim does not match ^[a-z][a-zA-Z0-9.\-]*$

deimjons commented 3 months ago

Hi all. I trying to foolow documentation and on first step I got error(((

error: Earthfile line 401:4 copy artifact: apply build +gtfs-get-mobilitydb: earthfile2llb for +gtfs-get-mobilitydb: parse target name +debian:bookworm-slim: target name debian:bookworm-slim does not match ^[a-z][a-zA-Z0-9.\-]*$

View logs at https://cloud.earthly.dev/builds/8ad964fb-460a-4b0f-936c-81e27318f66b

In general is it possible to build without proprietary tools such as earthly?

michaelkirk commented 3 months ago

Hm. I've never seen that error before.

In general is it possible to build without proprietary tools such as earthly?

I think some kind of build tooling is necessary to get all the bits and pieces you need for a self contained map stack.

I'm not sure what you mean by "proprietary" in this context - but last time I checked, the earthly client and local runner we use are all all open source (let me know if you know otherwise).

It looks like you're using earthly cloud - is that true? I've never used that before, so I'm not sure what kind of problems that might introduce. Can I ask what lead you to using that vs a self hosted runner?

deimjons commented 3 months ago

Thank you for the fast answer. I just follow this documentation: link

So, if I understand your answer correctly, it is possible to build everything with other tools. You mean this:

There is a script contributed by Santiago Crespo that will automatically deploy Headway as a systemd service on Debian, but it has not been widely tested yet. See contrib/DEBIAN_BUILD.md for details.

michaelkirk commented 3 months ago

No, I've never use the contrib build scripts, I use earthly.

Earthly has a local version. I think you are using the cloud version. I do not use the cloud version.

e.g. see the CI script which uses earthly: https://github.com/headwaymaps/headway/blob/main/.github/workflows/checks.yml#L20

deimjons commented 3 months ago

I make build locally. Let's describe my steps:

Clone repo
Install Earthly
trying to build Amsterdam on host machine with command: earthly -P +gtfs-enumerate --area="Amsterdam"

and here I got error:

headwaymaps % earthly -P +gtfs-enumerate --area="Amsterdam" Init 🚀 ————————————————————————————————————————————————————————————————————————————————
       buildkitd | Starting buildkit daemon as a docker container (earthly-buildkitd)...
       buildkitd | ...Done
Streaming logs to https://cloud.earthly.dev/builds/4d7ba9f1-d334-4452-83ea-f869c3a16d88

Build 🔧 ————————————————————————————————————————————————————————————————————————————————
          logbus | Setting organization "deimjons" and project ""
 +gtfs-enumerate | area=Amsterdam
 +gtfs-enumerate | --> FROM +base
 +gtfs-enumerate | area=Amsterdam
 +gtfs-enumerate | --> FROM +gtfs-base
      +gtfs-base | area=Amsterdam
      +gtfs-base | --> FROM +base
        python:3 | --> Load metadata python:3 linux/arm64
   +cache-buster | area=Amsterdam
   +cache-buster | --> FROM +base
debian:bookworm-slim | --> Load metadata debian:bookworm-slim linux/arm64 +gtfs-base | area=Amsterdam +gtfs-base | --> FROM python:3 +gtfs-base | [----------] 100% FROM python:3 +gtfs-base | area=Amsterdam +gtfs-base | cached --> RUN pip install requests +gtfs-base | area=Amsterdam +gtfs-base | cached --> WORKDIR /gtfs +gtfs-base | area=Amsterdam +gtfs-base | cached --> RUN mkdir /gtfs_feeds +cache-buster | area=Amsterdam +cache-buster | --> FROM debian:bookworm-slim +cache-buster | [----------] 100% FROM debian:bookworm-slim +cache-buster | area=Amsterdam +cache-buster | --> RUN --no-cache echo $(date +%Y-%m-%d) > todays_date +gtfs-enumerate | area=Amsterdam +gtfs-enumerate | cached --> COPY ./services/gtfs/filter_feeds.py /gtfs/ +gtfs-enumerate | area=Amsterdam +gtfs-enumerate | --> COPY +cache-buster/todays_date ./ +gtfs-enumerate | area=Amsterdam +gtfs-enumerate | --> mkdir /run/buildargs +gtfs-enumerate | area=Amsterdam +gtfs-enumerate | --> expandargs cat todays_date +gtfs-get-mobilitydb | area=Amsterdam cache_key=2024-03-25 +gtfs-get-mobilitydb | --> FROM +base Error: Earthfile line 401:4 copy artifact: apply build +gtfs-get-mobilitydb: earthfile2llb for +gtfs-get-mobilitydb: parse target name +debian:bookworm-slim: target name debian:bookworm-slim does not match ^[a-z][a-zA-Z0-9.-]*$ in github.com/headwaymaps/headway:main+gtfs-enumerate --cache_key= View logs at https://cloud.earthly.dev/builds/4d7ba9f1-d334-4452-83ea-f869c3a16d88

What I'm doing wrong?

michaelkirk commented 3 months ago

How did you install earthly?

This line is strange to me:

Streaming logs to https://cloud.earthly.dev/builds/4d7ba9f1-d334-4452-83ea-f869c3a16d88

It looks like you are using earthly cloud, which I do not use.

michaelkirk commented 3 months ago

Thank you for sharing your steps. I am able to reproduce your error now.

I don't regularly re-run the gtfs step, so it seems like something is broken with that step in particular. I'll take a look.

You can try skipping that step for now and everything should work except transit directions.

deimjons commented 3 months ago

I just follow earthly installation from this link.. https://cloud.earthly.dev/deimjons/tutorial/cli

How I should run this command: earthly -P +gtfs-enumerate --area="Amsterdam" to force earthly make local build?

ellenhp commented 3 months ago

edit: nevermind I missed that michael was able to repro this!

I think it does stream logs even if you build locally. I don't have a machine set up to do builds right now but if you introduce a syntax error in the earthfile do you get the same line in your logs @michaelkirk?

@deimjons This is a strange error that I haven't seen before either. Could you do two things: git status and post the output, just to make sure the working tree is clean. If something was modified somehow that could cause this. The second thing is run earthly prune --reset and then try again. Earthly is, in my experience, very good at invalidating caches correctly and not caching failed steps, but this will give you a clean slate to run the build again.

michaelkirk commented 3 months ago

Fixed in https://github.com/headwaymaps/headway/pull/333

I think it does stream logs even if you build locally.

I don't think that's true. I've never even set up an earthly cloud account.

I do see that they seem to be ratcheting up the money-machine. Under "Products" they have a link to "Earthly" ("earthly-core") and another link to "Earthly Cloud", but frustratingly they both lead you down the conversion funnel to a page with a big "Get Started" link that sends you to their cloud product. That's frustrating if accidental, and disheartening if intentional. But so it goes. It's hard to make a living on open source software.

BTW to use non-cloud earthly, see https://docs.earthly.dev/docs/misc/alt-installation for instructions. I'll update our own instructions to clarify this.

michaelkirk commented 3 months ago

Just to check @deimjons - did you go down the path of setting up an account on cloud.earthly.dev at some point? You shouldn't have to do that, and I'm sorry if our instructions lead you to do that. To be clear though, I think the older version of their site was a lot less hostile about tricking people into signing up for their cloud product.

deimjons commented 3 months ago

@ellenhp

earthly prune --reset - Also not helps..((

alex@MacBook-Air headwaymaps % earthly prune --reset buildkitd | Restarting buildkit daemon with reset command... buildkitd | ... Done alex@MacBook-Air headwaymaps % earthly -P +gtfs-enumerate --area="Amsterdam" Init 🚀 ————————————————————————————————————————————————————————————————————————————————
       buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
Streaming logs to https://cloud.earthly.dev/builds/b8856195-f662-4804-b30f-b86cf55f117f

Build 🔧 ————————————————————————————————————————————————————————————————————————————————
          logbus | Setting organization "deimjons" and project ""
 +gtfs-enumerate | area=Amsterdam
 +gtfs-enumerate | --> FROM +base
 +gtfs-enumerate | area=Amsterdam
 +gtfs-enumerate | --> FROM +gtfs-base
      +gtfs-base | area=Amsterdam
      +gtfs-base | --> FROM +base
        python:3 | --> Load metadata python:3 linux/arm64
   +cache-buster | area=Amsterdam
   +cache-buster | --> FROM +base
debian:bookworm-slim | --> Load metadata debian:bookworm-slim linux/arm64 +gtfs-base | area=Amsterdam +gtfs-base | --> FROM python:3 +gtfs-base | [----------] 100% FROM python:3 +cache-buster | area=Amsterdam +cache-buster | --> FROM debian:bookworm-slim +cache-buster | [----------] 100% FROM debian:bookworm-slim +gtfs-base | [4 ] 4% FROM python:3 +gtfs-base | [----------] 100% FROM python:3 +cache-buster | [----2 ] 42% FROM python:3 +cache-buster | [----------] 100% FROM debian:bookworm-slim +cache-buster | area=Amsterdam +cache-buster | --> RUN --no-cache echo $(date +%Y-%m-%d) > todays_date +gtfs-base | [----------] 100% FROM python:3 +gtfs-base | [----------] 100% FROM python:3 +gtfs-base | area=Amsterdam +gtfs-base | --> RUN pip install requests +gtfs-base | Collecting requests +gtfs-base | Downloading requests-2.31.0-py3-none-any.whl.metadata (4.6 kB) +gtfs-base | Collecting charset-normalizer<4,>=2 (from requests) +gtfs-base | Downloading charset_normalizer-3.3.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (33 kB) +gtfs-base | Collecting idna<4,>=2.5 (from requests) +gtfs-base | Downloading idna-3.6-py3-none-any.whl.metadata (9.9 kB) +gtfs-base | Collecting urllib3<3,>=1.21.1 (from requests) +gtfs-base | Downloading urllib3-2.2.1-py3-none-any.whl.metadata (6.4 kB) +gtfs-base | Collecting certifi>=2017.4.17 (from requests) +gtfs-base | Downloading certifi-2024.2.2-py3-none-any.whl.metadata (2.2 kB) +gtfs-base | Downloading requests-2.31.0-py3-none-any.whl (62 kB) +gtfs-base | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 5.8 MB/s eta 0:00:00 +gtfs-base | Downloading certifi-2024.2.2-py3-none-any.whl (163 kB) +gtfs-base | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 163.8/163.8 kB 20.1 MB/s eta 0:00:00
      +gtfs-base | Downloading charset_normalizer-3.3.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (137 kB)
      +gtfs-base |    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 137.3/137.3 kB 20.2 MB/s eta 0:00:00
      +gtfs-base | Downloading idna-3.6-py3-none-any.whl (61 kB)
      +gtfs-base |    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.6/61.6 kB 15.4 MB/s eta 0:00:00

      +gtfs-base | Downloading urllib3-2.2.1-py3-none-any.whl (121 kB)
      +gtfs-base |    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.1/121.1 kB 26.1 MB/s eta 0:00:00
      +gtfs-base | Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests
      +gtfs-base | Successfully installed certifi-2024.2.2 charset-normalizer-3.3.2 idna-3.6 requests-2.31.0 urllib3-2.2.1
      +gtfs-base | WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
      +gtfs-base | area=Amsterdam
      +gtfs-base | --> WORKDIR /gtfs
      +gtfs-base | area=Amsterdam
      +gtfs-base | --> RUN mkdir /gtfs_feeds
 +gtfs-enumerate | area=Amsterdam
 +gtfs-enumerate | --> COPY ./services/gtfs/filter_feeds.py /gtfs/
 +gtfs-enumerate | area=Amsterdam
 +gtfs-enumerate | --> COPY +cache-buster/todays_date ./
 +gtfs-enumerate | area=Amsterdam
 +gtfs-enumerate | --> mkdir /run/buildargs
 +gtfs-enumerate | area=Amsterdam
 +gtfs-enumerate | --> expandargs cat todays_date
+gtfs-get-mobilitydb | area=Amsterdam cache_key=2024-03-25 +gtfs-get-mobilitydb | --> FROM +base Error: Earthfile line 401:4 copy artifact: apply build +gtfs-get-mobilitydb: earthfile2llb for +gtfs-get-mobilitydb: parse target name +debian:bookworm-slim: target name debian:bookworm-slim does not match ^[a-z][a-zA-Z0-9.-]*$ in github.com/headwaymaps/headway:main+gtfs-enumerate --cache_key= View logs at https://cloud.earthly.dev/builds/b8856195-f662-4804-b30f-b86cf55f117f

michaelkirk commented 3 months ago

Just in case you missed it, this was fixed yesterday with #333

Could you update headway and try again?

deimjons commented 3 months ago

Thank you for the help. Now this step working correctly but I got error on step 3 here: https://github.com/headwaymaps/headway/blob/main/BUILD.md#build-procedure

1) In documentation is missing " after _earthly -P +build --area="Amsterdam" --transit_feeds="data/Amsterdam.gtsfeeds.csv

I think should be: earthly -P +build --area="Amsterdam" --transit_feeds="data/Amsterdam.gts_feeds.csv"

2) I got an error:

log output

> alex@MacBook-Air headwaymaps % earthly -P +build --area="Amsterdam" --transit_feeds="data/Amsterdam.gts_feeds.csv
> dquote> "
>  Init 🚀
> ————————————————————————————————————————————————————————————————————————————————
> 
>            buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
> 
> Streaming logs to https://cloud.earthly.dev/builds/0f6d1db1-76b7-41cb-9836-d05ffb99e508
> 
>  Build 🔧
> ————————————————————————————————————————————————————————————————————————————————
> 
>               logbus | Setting organization "deimjons" and project ""
>               +build | area=Amsterdam transit_feeds=data/Amsterdam.gts_feeds.csv
>               +build | 
>               +build | --> FROM +base
>               +build | area=Amsterdam transit_feeds=data/Amsterdam.gts_feeds.csv
>               +build | 
>               +build | --> BUILD +save
>                +save | area=Amsterdam countries= transit_feeds=data/Amsterdam.gts_feeds.csv
>                +save | 
>                +save | --> FROM +base
>              +images | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>              +images | 
>              +images | --> FROM +base
>                +save | area=Amsterdam countries= transit_feeds=data/Amsterdam.gts_feeds.csv
>                +save | 
>                +save | --> FROM +save-base
>           +save-base | area=Amsterdam countries= transit_feeds=data/Amsterdam.gts_feeds.csv
>           +save-base | 
>           +save-base | --> FROM +base
> debian:bookworm-slim | --> Load metadata debian:bookworm-slim linux/arm64
>                +save | area=Amsterdam countries= transit_feeds=data/Amsterdam.gts_feeds.csv
>                +save | 
>                +save | --> BUILD +save-extract
>     +otp-serve-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>     +otp-serve-image | 
>     +otp-serve-image | --> FROM +base
>              +images | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>              +images | 
>              +images | --> BUILD +travelmux-serve-image
> +valhalla-serve-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
> +valhalla-serve-image | 
> +valhalla-serve-image | --> FROM +base
>     +web-serve-image | area=Amsterdam branding= tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>     +web-serve-image | 
>     +web-serve-image | --> FROM +base
> +travelmux-serve-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
> +travelmux-serve-image | 
> +travelmux-serve-image | --> FROM +base
>        +save-extract | area=Amsterdam countries= transit_feeds=data/Amsterdam.gts_feeds.csv
>        +save-extract | 
>        +save-extract | --> FROM +base
>   +pelias-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>   +pelias-init-image | 
>   +pelias-init-image | --> FROM +base
>        +save-mbtiles | area=Amsterdam countries= transit_feeds=data/Amsterdam.gts_feeds.csv
>        +save-mbtiles | 
>        +save-mbtiles | --> FROM +base
> +valhalla-serve-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
> +valhalla-serve-image | 
> +valhalla-serve-image | --> FROM +valhalla-base-image
>     +otp-serve-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>     +otp-serve-image | 
>     +otp-serve-image | --> FROM +otp-base
>            +otp-base | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>            +otp-base | 
>            +otp-base | --> FROM +base
>        +save-extract | area=Amsterdam countries= transit_feeds=data/Amsterdam.gts_feeds.csv
>        +save-extract | 
>        +save-extract | --> FROM +save-base
>   +pelias-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>   +pelias-init-image | 
>   +pelias-init-image | --> FROM +downloader-base
> +valhalla-base-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
> +valhalla-base-image | 
> +valhalla-base-image | --> FROM +base
>     +build-travelmux | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>     +build-travelmux | 
>     +build-travelmux | --> FROM +base
>        +save-mbtiles | area=Amsterdam countries= transit_feeds=data/Amsterdam.gts_feeds.csv
>        +save-mbtiles | 
>        +save-mbtiles | --> FROM +save-base
>     +downloader-base | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>     +downloader-base | 
>     +downloader-base | --> FROM +base
>                nginx | --> Load metadata nginx linux/arm64
>           +save-base | area=Amsterdam countries= transit_feeds=data/Amsterdam.gts_feeds.csv
>           +save-base | 
>           +save-base | --> FROM debian:bookworm-slim
>             +extract | area=Amsterdam countries= transit_feeds=data/Amsterdam.gts_feeds.csv
>             +extract | 
>             +extract | --> FROM +base
> o/opentripplanner:2.3.0@sha256:630779e4b595462502b3813c1d6141da3e180d266d4a26371cc4ab6cb3390db0 | --> Load metadata opentripplanner/opentripplanner:2.3.0@sha256:630779e4b595462502b3813c1d6141da3e180d266d4a26371cc4ab6cb3390db0 linux/arm64
>             +extract | area=Amsterdam countries= transit_feeds=data/Amsterdam.gts_feeds.csv
>             +extract | 
>             +extract | --> FROM +downloader-base
> +planetiler-build-mbtiles | area=Amsterdam countries= transit_feeds=data/Amsterdam.gts_feeds.csv
> +planetiler-build-mbtiles | 
> +planetiler-build-mbtiles | --> FROM +base
>     +downloader-base | area=Amsterdam countries= transit_feeds=data/Amsterdam.gts_feeds.csv
>     +downloader-base | 
>     +downloader-base | --> FROM +base
>        rust:bookworm | --> Load metadata rust:bookworm linux/arm64
> g/g/d/valhalla:3.4.0@sha256:b6f20757c5a9d8bb432b53cb2923af36eb8908486d97fd1fdd114499a6d2a436 | --> Load metadata ghcr.io/gis-ops/docker-valhalla/valhalla:3.4.0@sha256:b6f20757c5a9d8bb432b53cb2923af36eb8908486d97fd1fdd114499a6d2a436 linux/arm64
> g/o/planetiler:0.7.0 | --> Load metadata ghcr.io/onthegomap/planetiler:0.7.0 linux/arm64
>           +save-base | [----------] 100% FROM debian:bookworm-slim
>           +save-base | area=Amsterdam countries= transit_feeds=data/Amsterdam.gts_feeds.csv
>           +save-base | 
>           +save-base | *cached* --> RUN apt-get update && apt-get install -y --no-install-recommends zip zstd && rm -rf /var/lib/apt/lists/*
> +tileserver-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
> +tileserver-init-image | 
> +tileserver-init-image | --> FROM +base
> +tileserver-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
> +tileserver-init-image | 
> +tileserver-init-image | --> FROM +downloader-base
> +valhalla-base-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
> +valhalla-base-image | 
> +valhalla-base-image | --> FROM ghcr.io/gis-ops/docker-valhalla/valhalla:3.4.0@sha256:b6f20757c5a9d8bb432b53cb2923af36eb8908486d97fd1fdd114499a6d2a436
>            +otp-base | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>            +otp-base | 
>            +otp-base | --> FROM opentripplanner/opentripplanner:2.3.0@sha256:630779e4b595462502b3813c1d6141da3e180d266d4a26371cc4ab6cb3390db0
> +valhalla-base-image | [----------] 100% FROM ghcr.io/gis-ops/docker-valhalla/valhalla:3.4.0@sha256:b6f20757c5a9d8bb432b53cb2923af36eb8908486d97fd1fdd114499a6d2a436
>            +otp-base | [----------] 100% FROM opentripplanner/opentripplanner:2.3.0@sha256:630779e4b595462502b3813c1d6141da3e180d266d4a26371cc4ab6cb3390db0
>     +downloader-base | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>     +downloader-base | 
>     +downloader-base | *cached* --> RUN apt-get update && apt-get install -y --no-install-recommends wget ca-certificates zstd && rm -rf /var/lib/apt/lists/*
>     +downloader-base | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>     +downloader-base | 
>     +downloader-base | *cached* --> RUN mkdir /data
>             +extract | area=Amsterdam countries= transit_feeds=data/Amsterdam.gts_feeds.csv
>             +extract | 
>             +extract | *cached* --> RUN apt-get update && apt-get install -y --no-install-recommends osmium-tool && rm -rf /var/lib/apt/lists/*
>             +extract | area=Amsterdam countries= transit_feeds=data/Amsterdam.gts_feeds.csv
>             +extract | 
>             +extract | *cached* --> WORKDIR /data
>             +extract | area=Amsterdam countries= transit_feeds=data/Amsterdam.gts_feeds.csv
>             +extract | 
>             +extract | *cached* --> COPY --if-exists Amsterdam.osm.pbf data.osm.pbf
>             +extract | area=Amsterdam countries= transit_feeds=data/Amsterdam.gts_feeds.csv
>             +extract | 
>             +extract | *cached* --> mkdir /run
>             +extract | area=Amsterdam countries= transit_feeds=data/Amsterdam.gts_feeds.csv
>             +extract | 
>             +extract | *cached* --> IF [ ! -f data.osm.pbf ]
>   +pelias-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>   +pelias-init-image | 
>   +pelias-init-image | *cached* --> RUN mkdir -p /app
>   +pelias-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>   +pelias-init-image | 
>   +pelias-init-image | *cached* --> COPY ./services/pelias/init* /app/
>   +pelias-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>   +pelias-init-image | 
>   +pelias-init-image | *cached* --> mkdir /run/buildargs
>   +pelias-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>   +pelias-init-image | 
>   +pelias-init-image | *cached* --> FOR ${tags}
> +tileserver-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
> +tileserver-init-image | 
> +tileserver-init-image | *cached* --> COPY ./services/tileserver/init.sh /app/init.sh
> +tileserver-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
> +tileserver-init-image | 
> +tileserver-init-image | *cached* --> mkdir /run/buildargs
> +tileserver-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
> +tileserver-init-image | 
> +tileserver-init-image | *cached* --> FOR ${tags}
>   +pelias-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>   +pelias-init-image | 
>   +pelias-init-image | --> SAVE IMAGE ghcr.io/headwaymaps/pelias-init:dev
>      +web-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>      +web-init-image | 
>      +web-init-image | --> FROM +base
>      +web-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>      +web-init-image | 
>      +web-init-image | --> FROM +downloader-base
> +tileserver-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
> +tileserver-init-image | 
> +tileserver-init-image | --> SAVE IMAGE ghcr.io/headwaymaps/tileserver-init:dev
>      +otp-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>      +otp-init-image | 
>      +otp-init-image | --> FROM +base
>      +otp-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>      +otp-init-image | 
>      +otp-init-image | --> FROM +downloader-base
>             +extract | area=Amsterdam countries= transit_feeds=data/Amsterdam.gts_feeds.csv
>             +extract | 
>             +extract | --> RUN wget -nv -U headway/1.0 -O data.osm.pbf "https://download.bbbike.org/osm/bbbike/${area}/${area}.osm.pbf"
>      +web-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>      +web-init-image | 
>      +web-init-image | *cached* --> COPY ./services/frontend/init.sh ./services/frontend/generate_config.sh /app/
>      +web-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>      +web-init-image | 
>      +web-init-image | *cached* --> mkdir /run/buildargs
>      +web-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>      +web-init-image | 
>      +web-init-image | *cached* --> FOR ${tags}
>      +otp-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>      +otp-init-image | 
>      +otp-init-image | *cached* --> COPY ./services/otp/init.sh /app/init.sh
>      +otp-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>      +otp-init-image | 
>      +otp-init-image | *cached* --> mkdir /run/buildargs
>      +otp-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>      +otp-init-image | 
>      +otp-init-image | *cached* --> FOR ${tags}
>      +web-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>      +web-init-image | 
>      +web-init-image | --> SAVE IMAGE ghcr.io/headwaymaps/headway-init:dev
>      +otp-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>      +otp-init-image | 
>      +otp-init-image | --> SAVE IMAGE ghcr.io/headwaymaps/opentripplanner-init:dev
> +valhalla-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
> +valhalla-init-image | 
> +valhalla-init-image | --> FROM +base
> +tileserver-serve-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
> +tileserver-serve-image | 
> +tileserver-serve-image | --> FROM +base
> +valhalla-init-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
> +valhalla-init-image | 
> +valhalla-init-image | --> FROM +valhalla-base-image
>         node:20-slim | --> Load metadata node:20-slim linux/arm64
>           +web-build | area=Amsterdam branding= tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>           +web-build | 
>           +web-build | --> FROM +base
>     +build-travelmux | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>     +build-travelmux | 
>     +build-travelmux | --> FROM rust:bookworm
> +travelmux-serve-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
> +travelmux-serve-image | 
> +travelmux-serve-image | *cached* --> RUN apt-get update && apt-get install -y --no-install-recommends libssl3 && rm -rf /var/lib/apt/lists/*
> +travelmux-serve-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
> +travelmux-serve-image | 
> +travelmux-serve-image | *cached* --> RUN adduser --disabled-login travelmux --gecos ""
> +travelmux-serve-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
> +travelmux-serve-image | 
> +travelmux-serve-image | *cached* --> WORKDIR /home/travelmux
>     +build-travelmux | [----------] 100% FROM rust:bookworm
>    +tileserver-build | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>    +tileserver-build | 
>    +tileserver-build | --> FROM +base
>           +web-build | area=Amsterdam branding= tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>           +web-build | 
>           +web-build | --> FROM node:20-slim
>           +web-build | [----------] 100% FROM node:20-slim
> +tileserver-serve-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
> +tileserver-serve-image | 
> +tileserver-serve-image | --> RUN npm install -g tileserver-gl-light
>           +web-build | area=Amsterdam branding= tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>           +web-build | 
>           +web-build | --> RUN yarn global add @quasar/cli
>    +tileserver-build | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>    +tileserver-build | 
>    +tileserver-build | *cached* --> COPY ./services/tileserver/assets/build_glyphs.js ./services/tileserver/assets/build_sprites.js ./services/tileserver/assets/package.json ./services/tileserver/assets/package-lock.json ./services/tileserver/assets/*.ttf /app/
>    +tileserver-build | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>    +tileserver-build | 
>    +tileserver-build | *cached* --> WORKDIR /app
>    +tileserver-build | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>    +tileserver-build | 
>    +tileserver-build | --> RUN npm install
>           +web-build | yarn global v1.22.19
>           +web-build | [1/4] Resolving packages...
>    +tileserver-build | npm WARN deprecated stable@0.1.8: Modern JS already guarantees Array#sort() is a stable sort, so this library is deprecated. See the compatibility table on MDN: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort#browser_compatibility
>    +tileserver-build | npm ERR! code 1
>    +tileserver-build | npm ERR! path /app/node_modules/fontnik
>    +tileserver-build | npm ERR! command failed
>    +tileserver-build | npm ERR! command sh -c node-gyp-build
>    +tileserver-build | npm ERR! gyp info it worked if it ends with ok
>    +tileserver-build | npm ERR! gyp info using node-gyp@10.0.1
>    +tileserver-build | npm ERR! gyp info using node@20.12.0 | linux | arm64
>    +tileserver-build | npm ERR! gyp ERR! find Python 
>    +tileserver-build | npm ERR! gyp ERR! find Python Python is not set from command line or npm configuration
>    +tileserver-build | npm ERR! gyp ERR! find Python Python is not set from environment variable PYTHON
>    +tileserver-build | npm ERR! gyp ERR! find Python checking if "python3" can be used
>    +tileserver-build | npm ERR! gyp ERR! find Python - executable path is ""
>    +tileserver-build | npm ERR! gyp ERR! find Python - "" could not be run
>    +tileserver-build | npm ERR! gyp ERR! find Python checking if "python" can be used
>    +tileserver-build | npm ERR! gyp ERR! find Python - executable path is ""
>    +tileserver-build | npm ERR! gyp ERR! find Python - "" could not be run
>    +tileserver-build | npm ERR! gyp ERR! find Python 
>    +tileserver-build | npm ERR! gyp ERR! find Python **********************************************************
>    +tileserver-build | npm ERR! gyp ERR! find Python You need to install the latest version of Python.
>    +tileserver-build | npm ERR! gyp ERR! find Python Node-gyp should be able to find and use Python. If not,
>    +tileserver-build | npm ERR! gyp ERR! find Python you can try one of the following options:
>    +tileserver-build | npm ERR! gyp ERR! find Python - Use the switch --python="/path/to/pythonexecutable"
>    +tileserver-build | npm ERR! gyp ERR! find Python (accepted by both node-gyp and npm)
>    +tileserver-build | npm ERR! gyp ERR! find Python - Set the environment variable PYTHON
>    +tileserver-build | npm ERR! gyp ERR! find Python - Set the npm configuration variable python:
>    +tileserver-build | npm ERR! gyp ERR! find Python npm config set python "/path/to/pythonexecutable"
>    +tileserver-build | npm ERR! gyp ERR! find Python For more information consult the documentation at:
>    +tileserver-build | npm ERR! gyp ERR! find Python https://github.com/nodejs/node-gyp#installation
>    +tileserver-build | npm ERR! gyp ERR! find Python **********************************************************
>    +tileserver-build | npm ERR! gyp ERR! find Python 
>    +tileserver-build | npm ERR! gyp ERR! configure error 
>    +tileserver-build | npm ERR! gyp ERR! stack Error: Could not find any Python installation to use
>    +tileserver-build | npm ERR! gyp ERR! stack at PythonFinder.fail (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:306:11)
>    +tileserver-build | npm ERR! gyp ERR! stack at PythonFinder.findPython (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:164:17)
>    +tileserver-build | npm ERR! gyp ERR! stack at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
>    +tileserver-build | npm ERR! gyp ERR! stack at async configure (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:23:18)
>    +tileserver-build | npm ERR! gyp ERR! stack at async run (/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js:81:18)
>    +tileserver-build | npm ERR! gyp ERR! System Linux 6.6.16-linuxkit
>    +tileserver-build | npm ERR! gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
>    +tileserver-build | npm ERR! gyp ERR! cwd /app/node_modules/fontnik
>    +tileserver-build | npm ERR! gyp ERR! node -v v20.12.0
>    +tileserver-build | npm ERR! gyp ERR! node-gyp -v v10.0.1
>    +tileserver-build | npm ERR! gyp ERR! not ok
> 
>    +tileserver-build | npm ERR! A complete log of this run can be found in: /root/.npm/_logs/2024-03-29T20_49_45_071Z-debug-0.log
>    +tileserver-build | ERROR Earthfile line 713:4
>    +tileserver-build |       The command
>    +tileserver-build |           RUN npm install
>    +tileserver-build |       did not complete successfully. Exit code 1
> 
> ================================== ❌ FAILURE ===================================
> 
>    +tileserver-build *failed* | Repeating the failure error...
>    +tileserver-build *failed* | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
>    +tileserver-build *failed* | 
>    +tileserver-build *failed* | --> RUN npm install
>    +tileserver-build *failed* | npm WARN deprecated stable@0.1.8: Modern JS already guarantees Array#sort() is a stable sort, so this library is deprecated. See the compatibility table on MDN: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/sort#browser_compatibility
>    +tileserver-build *failed* | npm ERR! code 1
>    +tileserver-build *failed* | npm ERR! path /app/node_modules/fontnik
>    +tileserver-build *failed* | npm ERR! command failed
>    +tileserver-build *failed* | npm ERR! command sh -c node-gyp-build
>    +tileserver-build *failed* | npm ERR! gyp info it worked if it ends with ok
>    +tileserver-build *failed* | npm ERR! gyp info using node-gyp@10.0.1
>    +tileserver-build *failed* | npm ERR! gyp info using node@20.12.0 | linux | arm64
>    +tileserver-build *failed* | npm ERR! gyp ERR! find Python 
>    +tileserver-build *failed* | npm ERR! gyp ERR! find Python Python is not set from command line or npm configuration
>    +tileserver-build *failed* | npm ERR! gyp ERR! find Python Python is not set from environment variable PYTHON
>    +tileserver-build *failed* | npm ERR! gyp ERR! find Python checking if "python3" can be used
>    +tileserver-build *failed* | npm ERR! gyp ERR! find Python - executable path is ""
>    +tileserver-build *failed* | npm ERR! gyp ERR! find Python - "" could not be run
>    +tileserver-build *failed* | npm ERR! gyp ERR! find Python checking if "python" can be used
>    +tileserver-build *failed* | npm ERR! gyp ERR! find Python - executable path is ""
>    +tileserver-build *failed* | npm ERR! gyp ERR! find Python - "" could not be run
>    +tileserver-build *failed* | npm ERR! gyp ERR! find Python 
>    +tileserver-build *failed* | npm ERR! gyp ERR! find Python **********************************************************
>    +tileserver-build *failed* | npm ERR! gyp ERR! find Python You need to install the latest version of Python.
>    +tileserver-build *failed* | npm ERR! gyp ERR! find Python Node-gyp should be able to find and use Python. If not,
>    +tileserver-build *failed* | npm ERR! gyp ERR! find Python you can try one of the following options:
>    +tileserver-build *failed* | npm ERR! gyp ERR! find Python - Use the switch --python="/path/to/pythonexecutable"
>    +tileserver-build *failed* | npm ERR! gyp ERR! find Python (accepted by both node-gyp and npm)
>    +tileserver-build *failed* | npm ERR! gyp ERR! find Python - Set the environment variable PYTHON
>    +tileserver-build *failed* | npm ERR! gyp ERR! find Python - Set the npm configuration variable python:
>    +tileserver-build *failed* | npm ERR! gyp ERR! find Python npm config set python "/path/to/pythonexecutable"
>    +tileserver-build *failed* | npm ERR! gyp ERR! find Python For more information consult the documentation at:
>    +tileserver-build *failed* | npm ERR! gyp ERR! find Python https://github.com/nodejs/node-gyp#installation
>    +tileserver-build *failed* | npm ERR! gyp ERR! find Python **********************************************************
>    +tileserver-build *failed* | npm ERR! gyp ERR! find Python 
>    +tileserver-build *failed* | npm ERR! gyp ERR! configure error 
>    +tileserver-build *failed* | npm ERR! gyp ERR! stack Error: Could not find any Python installation to use
>    +tileserver-build *failed* | npm ERR! gyp ERR! stack at PythonFinder.fail (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:306:11)
>    +tileserver-build *failed* | npm ERR! gyp ERR! stack at PythonFinder.findPython (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:164:17)
>    +tileserver-build *failed* | npm ERR! gyp ERR! stack at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
>    +tileserver-build *failed* | npm ERR! gyp ERR! stack at async configure (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/configure.js:23:18)
>    +tileserver-build *failed* | npm ERR! gyp ERR! stack at async run (/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js:81:18)
>    +tileserver-build *failed* | npm ERR! gyp ERR! System Linux 6.6.16-linuxkit
>    +tileserver-build *failed* | npm ERR! gyp ERR! command "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
>    +tileserver-build *failed* | npm ERR! gyp ERR! cwd /app/node_modules/fontnik
>    +tileserver-build *failed* | npm ERR! gyp ERR! node -v v20.12.0
>    +tileserver-build *failed* | npm ERR! gyp ERR! node-gyp -v v10.0.1
>    +tileserver-build *failed* | npm ERR! gyp ERR! not ok
> 
>    +tileserver-build *failed* | npm ERR! A complete log of this run can be found in: /root/.npm/_logs/2024-03-29T20_49_45_071Z-debug-0.log
>    +tileserver-build *failed* | ERROR Earthfile line 713:4
>    +tileserver-build *failed* |       The command
>    +tileserver-build *failed* |           RUN npm install
>    +tileserver-build *failed* |       did not complete successfully. Exit code 1
> +valhalla-base-image | [          ]   0% FROM ghcr.io/gis-ops/docker-valhalla/valhalla:3.4.0@sha256:b6f20757c5a9d8bb432b53cb2923af36eb8908486d97fd1fdd114499a6d2a43[6         ]   6% FROM opentripplanner/opentripplanner:2.3.0@sha256:630779e4b595462502b3813c1d6141da3e180d266d4a26371cc4ab6cb3390db0        [------2   ]  62% FROM opentripplanner/opentripplanner:2.3.0@sha256:630779e4b595462502b3813c1d6141da3e180d266d4a26371cc4ab6cb3390db0        [6         ]   6% FROM opentripplanner/opentripplanner:2.3.0@sha256:630779e4b595462502b3813c1d6141da3e180d266d4a26371cc4ab6cb3390db0        [----------] 100% FROM ghcr.io/gis-ops/docker-valhalla/valhalla:3.4.0@sha256:b6f20757c5a9d8bb432b53cb2923af36eb8908486d97fd1fdd114499a6d2a436
> +valhalla-base-image | area=Amsterdam tags=dev transit_feeds=data/Amsterdam.gts_feeds.csv
> +valhalla-base-image | 
> +valhalla-base-image | --> WORKDIR /tiles
> 
> Help: To debug your build, you can use the --interactive (-i) flag to drop into a shell of the failing RUN step: "earthly -i -P +build --area=Amsterdam --transit_feeds=data/Amsterdam.gts_feeds.csv\n"
> 
> View logs at https://cloud.earthly.dev/builds/0f6d1db1-76b7-41cb-9836-d05ffb99e508

michaelkirk commented 3 months ago

I've seen a similar error while attempting to build on aarch64 (mac m1), but I typically use amd64 linux.

What arch are you building on?

deimjons commented 3 months ago

MacBook air 15 m2

michaelkirk commented 3 months ago

Sorry to say I don't have an immediate solution for building on mac. Because it's not on my own critical path, I haven't prioritized spending time fixing the mac build. I'd be happy to merge a PR if someone has a proposal.

deimjons commented 3 months ago

ok, I will get back to you and try again when I buy the server on Linux. Will you plan to simplify the building process? Maybe some single container for build (with settings based on the environment variables + a couple of containers that you run to use services).

I have set up and integrated more than 80 different services with more than 150 containers in my infrastructure. But this set of containers seems the most confusing to set up and integrate with existing infrastructure. To have more users, for example, guys who like self-hosted and host containers over Unraid or TrueNas or some Synology solutions, the process needs to be simplified and the number of containers reduced. Mb integrates build functionality inside containers that you run to use services. Any way thank yo for this amazing work. I hope one day I will be able to use it.

michaelkirk commented 3 months ago

The reality is it takes a lot of software to build and host a full map stack, so we have a lot of dependencies, and with those dependencies come their complications. Simplifying the build and improving the documentation is definitely an ongoing goal, so thanks for the feedback.

headwaymaps / headway

error: Earthfile line 401:4 copy artifact: apply build +gtfs-get-mobilitydb: earthfile2llb for +gtfs-get-mobilitydb: parse target name +debian:bookworm-slim: target name debian:bookworm-slim does not match ^[a-z][a-zA-Z0-9.\-]*$ #331