232-ingest-process-monitoring-seems-to-be-not-working

[randytpierce]

This makes the scraper work with the new log file format. It also accounts for promql being an architecture specific binary that must be installed on the import platofrm (it is installed in the container image but not necessarily when running the import as a script). This exercise reinforces the need to migrate the metrics into the code somehow, although that won't be a trivial exercise.

[ian-noaa]

A couple questions:

• Should we go ahead and remove the promql binary from the repo? It's architecture specific so will break on Apple Silicon/ARM dev laptops. It may be enough to test for a promql binary and exit with a warning to go install it if it's not found.
• I don't think the container has promql installed on PATH so this will break the container. We aren't currently using the container as far as I know but it'd be nice to switch to.

[randytpierce]

Yes, I believed I had done that on the last PR but if I missed it we definitely should. randy

On Wed, Jan 24, 2024 at 10:03 AM Ian McGinnis @.***> wrote:

A couple questions:

• Should we go ahead and remove the promql binary from the repo? It's architecture specific so will break on Apple Silicon/ARM dev laptops. It may be enough to test for a promql binary and exit with a warning to go install it if it's not found.
• I don't think the container has promql installed on PATH so this will break the container. We aren't currently using the container as far as I know but it'd be nice to switch to.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-GSL/VxIngest/pull/301#issuecomment-1908557080, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGDVQPRYDITY4GXHZBIUSVTYQE5FPAVCNFSM6AAAAABCFY6ORWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBYGU2TOMBYGA . You are receiving this because you were assigned.Message ID: @.***>

-- Randy Pierce

[ian-noaa]

Gotcha. It looks like promql was removed earlier. Sorry I missed that.

Otherwise, the import container image doesn't have the promql cli tool installed so we should update the docker/import/Dockerfile to make sure it's installed there. It looks like we should be able to get prebuilt binaries from this project: https://github.com/nalbury/promql-cli/releases/tag/v0.3.0.

I could see installing it the same way we do the Couchbase tooling: https://github.com/NOAA-GSL/VxIngest/blob/38e7bb58bea4fca7a3f9213260c558eddf600896/docker/import/Dockerfile#L34-L36

[randytpierce]

Oh wait, I had included the linux x86 64 image in the repo because that is the container architecture. Shucks! It is just a binary, so how do we handle the install to the container? randy

On Wed, Jan 24, 2024 at 10:25 AM Ian McGinnis @.***> wrote:

Gotcha. It looks like promql was removed earlier. Sorry I missed that.

Otherwise, the import container image doesn't have the promql cli tool installed so we should update the docker/import/Dockerfile to make sure it's installed there. It looks like we should be able to get prebuilt binaries from this project: https://github.com/nalbury/promql-cli/releases/tag/v0.3.0.

I could see installing it the same way we do the Couchbase tooling: https://github.com/NOAA-GSL/VxIngest/blob/38e7bb58bea4fca7a3f9213260c558eddf600896/docker/import/Dockerfile#L34-L36

— Reply to this email directly, view it on GitHub https://github.com/NOAA-GSL/VxIngest/pull/301#issuecomment-1908595787, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGDVQPX26LRBPQRE6XYHZM3YQE7ZLAVCNFSM6AAAAABCFY6ORWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBYGU4TKNZYG4 . You are receiving this because you were assigned.Message ID: @.***>

-- Randy Pierce

[ian-noaa]

I'd handle it like in the example I linked for Couchbase. It uses uname - m to get the architecture - though I'd need to verify it gets amd64 and arm64 as needed. For promql, the URLs are like: https://github.com/nalbury/promql-cli/releases/download/v0.3.0/promql-v0.3.0-linux-amd64.tar.gz. So we'd do:

ENV PROMQL_VERSION=v0.3.0
RUN curl -L https://github.com/nalbury/promql-cli/releases/download/${PROMQL_VERSION}/promql-${PROMQL_VERSION}-linux-$(uname -m).tar.gz | tar xz --directory /usr/local 

[randytpierce]

I added the promql install to the Dockerfile, Good catch!

[github-actions[bot]]

Package	Line Rate	Branch Rate	Health
vxingest	33%	42%	❌
vxingest.builder_common	29%	17%	❌
vxingest.ctc_to_cb	12%	1%	❌
vxingest.grib2_to_cb	13%	1%	❌
vxingest.netcdf_to_cb	13%	1%	❌
vxingest.partial_sums_to_cb	12%	1%	❌
vxingest.utilities	30%	32%	❌
Summary	16% (488 / 2989)	6% (40 / 684)	❌