There was an error making API call

[dbluxo]

Hello,

does the exporter have to run on every Artifactory host or can the external Web URL (e.g. https://artifactory.my-domain.com/artifactory) also be used as ARTI_SCRAPE_URI? Anyway, I get this error message using the Web URL:

level=info ts=2020-06-30T08:31:09.741Z caller=artifactory_exporter.go:30 msg="Starting artifactory_exporter" version="(version=master, branch=master, revision=48827804e60076fc770845154044b29395f098e7)"
level=info ts=2020-06-30T08:31:09.741Z caller=artifactory_exporter.go:31 msg="Build context" context="(go=go1.13.12, user=, date=2020-06-24T18:30:32Z)"
level=info ts=2020-06-30T08:31:09.741Z caller=artifactory_exporter.go:32 msg="Listening on address" address=:9531
level=debug ts=2020-06-30T08:31:13.372Z caller=system.go:66 msg="Fetching license stats"
level=debug ts=2020-06-30T08:31:13.373Z caller=utils.go:21 msg="Fetching http" path=https://artifactory.my-domain.com/artifactory/api/system/license
level=debug ts=2020-06-30T08:31:13.924Z caller=security.go:23 msg="Fetching users stats"
level=debug ts=2020-06-30T08:31:13.925Z caller=utils.go:21 msg="Fetching http" path=https://artifactory.my-domain.com/artifactory/api/security/users
level=error ts=2020-06-30T08:31:14.223Z caller=utils.go:50 msg="There was an error making API call" endpoint=https://artifactory.my-domain.com/artifactory/api/security/users err="[map[message:Error while trying to authenticate user 'techuser-monitoring'. status:401]]" status=(MISSING)
level=error ts=2020-06-30T08:31:14.224Z caller=security.go:43 msg="Couldn't scrape Artifactory when fetching security/users" err="[map[message:Error while trying to authenticate user 'techuser-monitoring'. status:401]]"

[ChaotX]

If I run the exporter in docker with localhost uri: docker run --rm --name prometheus_exporter --network host --env-file=$(pwd)/env_file peimanja/artifactory_expoer:latest --artifactory.scrape-uri="localhost/artifactory" --log.level=debug I see no error in the log, but on the /metrics I see increasing api error counter: "artifactory_exporter_total_api_errors 14"

If I run with web url: docker run --rm --name prometheus_exporter --network host --env-file=$(pwd)/env_file peimanja/artifactory_expoer:latest --artifactory.scrape-uri="https://XXXXX.XXXXX.XXXXX.com/artifactory" --log.level=debug I receive error:

level=debug ts=2020-06-30T11:03:06.564Z caller=system.go:66 msg="Fetching license stats" level=debug ts=2020-06-30T11:03:06.564Z caller=utils.go:21 msg="Fetching http" path=https://XXXXX.XXXXX.XXXXX.com/artifactory/api/system/license level=error ts=2020-06-30T11:03:07.134Z caller=utils.go:50 msg="There was an error making API call" endpoint=https://XXXXX.XXXXX.XXXXX.com/artifactory/api/system/license err="[map[message:Bad credentials status:401]]" status=(MISSING)

Or similar error with api-key:

level=debug ts=2020-06-30T11:05:51.564Z caller=system.go:66 msg="Fetching license stats" level=debug ts=2020-06-30T11:05:51.564Z caller=utils.go:21 msg="Fetching http" path=https://XXXXX.XXXXX.XXXXX.com/artifactory/api/system/license level=error ts=2020-06-30T11:05:51.576Z caller=utils.go:50 msg="There was an error making API call" endpoint=https://XXXXX.XXXXX.XXXXX.com/artifactory/api/system/license err="[map[message:Bad props auth token: basictoken=XXXXXXXXXXXXXXXXXXXXXXX status:401]]" status=(MISSING)

I even tried the older v0.8.0 image, no success and same results. v0.7.0 image only prints

level=debug ts=2020-06-30T11:09:51.564Z caller=system.go:51 msg="Fetching license stats" level=debug ts=2020-06-30T11:09:51.564Z caller=collector.go:163 msg="Fetching http" path=https://XXXXX.XXXXX.XXXXX.com/artifactory/api/system/license level=error ts=2020-06-30T11:09:51.599Z caller=collector.go:198 msg="Can't scrape Artifactory when fetching system/license" err="HTTP status 401"

I checked the https://XXXXX.XXXXX.XXXXX.com/artifactory/api/system/license with the username and password and it works in a browser.

I tried the container without host network option and even tried with binary release too.

[dbluxo]

Strange, I have executed the binary locally on my computer and it works from the second request on.

The first request always fails with:

level=error ts=2020-06-30T13:22:28.146Z caller=utils.go:53 msg="There was an error making API call" endpoint=https://artifactory.my-domain.com/artifactory/api/system/license err="Error while trying to authenticate user 'techuser-monitoring'." status=401

From the second request on I get a lot of metrics!

Why does it work locally but not with the Docker Image? 🤔

[peimanja]

@dbluxo @ChaotX Please make sure the user has admin access as some of the endpoints requires admin rights. The exporter is meant to run along with each node as there are some node specific metrics such a license expiary date. There might be a weird behaviour on autheentication if pointing to a LB but I will test that later on. for the purpose of testing when running on docker, just make sure your username/password are setting properly as environment variable.

docker run --name prometheus_exporter --network host --env ARTI_USERNAME=**** --env ARTI_PASSWORD=*** peimanja/artifactory_expoer:latest --artifactory.scrape-uri="https://<Instance-IP>/artifactory" --log.level=debug

Please also let me know which version of Artifactory you are running.

[dbluxo]

That was my mistake, I messed up the env file for Docker. It works now, I ran it all night. Funny, I'm still getting a 401 every five minutes:

level=error ts=2020-07-01T03:41:22.959Z caller=utils.go:101 msg="There was an error making API call" endpoint=https://artifactory.my-domain.com/artifactory/api/search/aql  err="Error while trying to authenticate user 'techuser-monitoring'." status=401
level=error ts=2020-07-01T03:46:52.441Z caller=utils.go:53 msg="There was an error making API call" endpoint=https://artifactory.my-domain.com/artifactory/api/system/license  err="Error while trying to authenticate user 'techuser-monitoring'." status=401
level=error ts=2020-07-01T03:51:52.842Z caller=utils.go:101 msg="There was an error making API call" endpoint=https://artifactory.my-domain.com/artifactory/api/search/aql  err="Error while trying to authenticate user 'techuser-monitoring'." status=401
level=error ts=2020-07-01T03:57:22.346Z caller=utils.go:53 msg="There was an error making API call" endpoint=https://artifactory.my-domain.com/artifactory/api/system/license  err="Error while trying to authenticate user 'techuser-monitoring'." status=401
level=error ts=2020-07-01T04:02:22.888Z caller=utils.go:101 msg="There was an error making API call" endpoint=https://artifactory.my-domain.com/artifactory/api/search/aql  err="Error while trying to authenticate user 'techuser-monitoring'." status=401
level=error ts=2020-07-01T04:07:52.399Z caller=utils.go:53 msg="There was an error making API call" endpoint=https://artifactory.my-domain.com/artifactory/api/system/license  err="Error while trying to authenticate user 'techuser-monitoring'." status=401
level=error ts=2020-07-01T04:12:52.770Z caller=utils.go:101 msg="There was an error making API call" endpoint=https://artifactory.my-domain.com/artifactory/api/search/aql  err="Error while trying to authenticate user 'techuser-monitoring'." status=401

Metrics:

artifactory_exporter_total_api_errors 108
artifactory_exporter_total_scrapes 1112

Can this be related to the possible LB problem you mentioned?

(Artifactory version 6.11.7)

[peimanja]

Possible. So please try that if it is possible to point to each node and let’s see if it is realted to that.

Can you also confirm that monitoring user is admin?

[ChaotX]

I had a typo in the env_file. The exporter works now.

[dbluxo]

So please try that if it is possible to point to each node and let’s see if it is realted to that.

I'll try that.

Can you also confirm that monitoring user is admin?

Yes.

[ilvidel]

Hello @peimanja ,

I think I'm facing a similar issue. I'm using the admin user, with the correct password, no typos, but all I get is this:

$ docker run -e ARTI_USERNAME=admin -e ARTI_PASSWORD=***** -p 9531:9531 peimanja/artifactory_exporter --log.level=debug
level=info ts=2021-05-07T11:13:44.009Z caller=artifactory_exporter.go:30 msg="Starting artifactory_exporter" version="(version=master, branch=master, revision=7528200b453bc35480c1ed62f4cb01b671886f03)"
level=info ts=2021-05-07T11:13:44.009Z caller=artifactory_exporter.go:31 msg="Build context" context="(go=go1.15.7, user=, date=2021-02-03T18:54:26Z)"
level=info ts=2021-05-07T11:13:44.009Z caller=artifactory_exporter.go:32 msg="Listening on address" address=:9531

level=debug ts=2021-05-07T11:13:48.429Z caller=system.go:66 msg="Fetching license stats"
level=debug ts=2021-05-07T11:13:48.429Z caller=utils.go:21 msg="Fetching http" path=http://localhost:8081/artifactory/api/system/license

$ curl -s localhost:9531/metrics | grep -v '#' | grep artifactory
artifactory_exporter_build_info{branch="master",goversion="go1.15.7",revision="7528200b453bc35480c1ed62f4cb01b671886f03",version="master"} 1
artifactory_exporter_json_parse_failures 0
artifactory_exporter_total_api_errors 1
artifactory_exporter_total_scrapes 1
artifactory_up 0

artifactory_exporter_total_api_errors gets increased every time I curl, but there's no indication as to what is going on.

I tried manually reaching the license URL, and it worked:

 $ curl -s http://localhost:8081/artifactory/api/system/license -u admin
Enter host password for user 'admin':
{
  "type" : "Community Edition for C/C++",
  "validThrough" : "",
  "licensedTo" : ""
}

Can you please advice? Thanks

[ilvidel]

Interesting. I found it's not working with my local Artifactory, but it does work with a remote one (tweaking the scrape uri)