Unable to use jiralib with Jira cloud and API Token

[opie4624]

Somewhat similar to #337, I'm unable to use an API token. What I've determined is the Authorization header created by requests is what's faulty. If I call an identical curl command manually using the -u parameter, it works fine. What's particularly interesting is that the authorization headers being generated by requests compared to the one generated by curl seem to be identical; the only change seems to be the ordering: curl seems to put it after cookies, while the requests version puts it before.

(Newlines in below commandlines are only to improve readability.)

This one (roughly as generated by requests) doesn't work:

curl 
  -v --location
  --cookie /Users/opie4624/.spacemacs/.cache/request/curl-cookie-jar
  --cookie-jar /Users/opie4624/.spacemacs/.cache/request/curl-cookie-jar
  --include --compressed --request POST
  --header 'Authorization: Basic b3BpZTQ2MjRAZXhhbXBsZS5jb206bm90LWEtcmVhbC1hcGktdG9rZW4='
  --header 'Content-Type: application/json' 
  --url https://my-domain.atlassian.net:443/rest/api/2/search
  -d '{"jql":"assignee = currentUser() and resolution = unresolved ORDER BY  priority DESC, created ASC","maxResults":100}'

This one works fine:

curl
  -v --location
  --cookie /Users/opie4624/.spacemacs/.cache/request/curl-cookie-jar
  --cookie-jar /Users/opie4624/.spacemacs/.cache/request/curl-cookie-jar
  --include --compressed --request POST
  -u 'opie4624@example.com:not-a-real-api-token' 
  --header 'Content-Type: application/json'
  --url https://my-domain.atlassian.net:443/rest/api/2/search
  -d '{"jql":"assignee = currentUser() and resolution = unresolved ORDER BY  priority DESC, created ASC","maxResults":100}'

[ahungry]

https://stackoverflow.com/questions/37865875/stopping-curl-from-sending-authorization-header-on-302-redirect

I wonder if something like that could be related?

When you hit your domain, are you receiving any sort of redirect? What version of curl are you using?

If there was a different default behavior between manually set header forwarding and the --user (-u) flag forwarding, it might cause this issue?

In the verbose output, do you see any matches for Authorization that differ between the two calls? (something like curl ... 2>&1 | grep Authorization - obviously don't paste verbatim :smile: ) - like listed multiple times in one call, and less times in the other?

[opie4624]

Here's what's showing up in Emacs from the requests debug output:

[debug] request--curl: --silent --location --cookie /Users/opie4624/.spacemacs/.cache/request/curl-cookie-jar --cookie-jar /Users/opie4624/.spacemacs/.cache/request/curl-cookie-jar --include --write-out \n(:num-redirects %{num_redirects} :url-effective "%{url_effective}") --compressed --request POST --header Authorization: Basic b3BpZTQ2MjRAZXhhbXBsZS5jb206bm90LWEtcmVhbC1hcGktdG9rZW4 --header Content-Type: application/json --url https://my-domain.atlassian.net:443/rest/api/2/search --data-binary @-
Fetching issues...
[debug] request--curl-callback: event finished

[debug] request--callback: UNPARSED
HTTP/2 400 
date: Wed, 27 Sep 2023 18:48:33 GMT
content-type: application/json;charset=UTF-8
server: AtlassianEdge
timing-allow-origin: *
x-arequestid: 8257bb7575dfe0d4477d80003a539012
x-seraph-loginreason: AUTHENTICATED_FAILED
cache-control: no-cache, no-store, no-transform
server-timing: filter-workcontext;dur=81, filter-frontend-router;dur=51, filter-request-papi;dur=81, sql;dur=4
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
atl-traceid: bf84a8a725a565c6
report-to: {"endpoints": [{"url": "https://dz8aopenkvv6s.cloudfront.net"}], "group": "endpoint-1", "include_subdomains": true, "max_age": 600}
nel: {"failure_fraction": 0.001, "include_subdomains": true, "max_age": 600, "report_to": "endpoint-1"}
strict-transport-security: max-age=63072000; includeSubDomains; preload

{"errorMessages":["Field 'assignee' does not exist or this field cannot be viewed by anonymous users.","Field 'resolution' does not exist or this field cannot be viewed by anonymous users.","Not able to sort using field 'priority'."],"warningMessages":[]}
[error] request--callback: peculiar error: 400
[debug] request--callback: executing error

"JIRA_ERROR - see your *Messages* buffer for more details."

"JIRA_ERROR REQUEST: "

"/rest/api/2/search"

(:type "POST" :data "{\"jql\":\"assignee = currentUser() and resolution = unresolved ORDER BY\\n  priority DESC, created ASC\",\"maxResults\":100}")

"JIRA_ERROR RESPONSE: "

((errorMessages . ["Field 'assignee' does not exist or this field cannot be viewed by anonymous users." "Field 'resolution' does not exist or this field cannot be viewed by anonymous users." "Not able to sort using field 'priority'."]) (warningMessages . []))

Here's the curl verbose log portion from a successful connection:

*   Trying 104.192.138.12:443...
* Connected to my-domain.atlassian.net (104.192.138.12) port 443 (#0)
* ALPN: offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Atlassian, Inc.; CN=*.atlassian.net
*  start date: Nov  7 00:00:00 2022 GMT
*  expire date: Dec  7 23:59:59 2023 GMT
*  subjectAltName: host "my-domain.atlassian.net" matched cert's "*.atlassian.net"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
* using HTTP/2
* Server auth using Basic with user 'opie4624@example.com'
* h2 [:method: POST]
* h2 [:scheme: https]
* h2 [:authority: my-domain.atlassian.net]
* h2 [:path: /rest/api/2/search]
* h2 [authorization: Basic b3BpZTQ2MjRAZXhhbXBsZS5jb206bm90LWEtcmVhbC1hcGktdG9rZW4]
* h2 [user-agent: curl/8.1.2]
* h2 [accept: */*]
* h2 [accept-encoding: deflate, gzip]
* h2 [cookie: atlassian.xsrf.token=aee68df5cfe7cc782c1caaf5446d80c20c415515_lout]
* h2 [content-type: application/json]
* h2 [content-length: 116]
* Using Stream ID: 1 (easy handle 0x125013400)
> POST /rest/api/2/search HTTP/2
> Host: my-domain.atlassian.net
> Authorization: Basic b3BpZTQ2MjRAZXhhbXBsZS5jb206bm90LWEtcmVhbC1hcGktdG9rZW4
> User-Agent: curl/8.1.2
> Accept: */*
> Accept-Encoding: deflate, gzip
> Cookie: atlassian.xsrf.token=aee68df5cfe7cc782c1caaf5446d80c20c415515_lout
> Content-Type: application/json
> Content-Length: 116
>
* We are completely uploaded and fine
< HTTP/2 200
HTTP/2 200
< date: Wed, 27 Sep 2023 20:51:53 GMT
date: Wed, 27 Sep 2023 20:51:53 GMT
< content-type: application/json;charset=UTF-8
content-type: application/json;charset=UTF-8
< server: AtlassianEdge
server: AtlassianEdge
< timing-allow-origin: *
timing-allow-origin: *
< x-arequestid: 24fd8c60e41d19713b9b1ddedd9933f3
x-arequestid: 24fd8c60e41d19713b9b1ddedd9933f3
* Replaced cookie atlassian.xsrf.token="6170bb1dabd9cf4c1fe692f883669c77f6f8c653_lin" for domain permiso-io.atlassian.net, path /, expire 0
< set-cookie: atlassian.xsrf.token=6170bb1dabd9cf4c1fe692f883669c77f6f8c653_lin; Path=/; SameSite=None; Secure
set-cookie: atlassian.xsrf.token=6170bb1dabd9cf4c1fe692f883669c77f6f8c653_lin; Path=/; SameSite=None; Secure
< x-aaccountid: 6215264c2d057500723f099e
x-aaccountid: 6215264c2d057500723f099e
< cache-control: no-cache, no-store, no-transform
cache-control: no-cache, no-store, no-transform
< server-timing: filter-workcontext;dur=1069, filter-frontend-router;dur=1021, mcache-client;dur=25, issue-search;dur=75, filter-request-papi;dur=1070, sql;dur=239
server-timing: filter-workcontext;dur=1069, filter-frontend-router;dur=1021, mcache-client;dur=25, issue-search;dur=75, filter-request-papi;dur=1070, sql;dur=239
< vary: Accept-Encoding
vary: Accept-Encoding
< content-encoding: gzip
content-encoding: gzip
< x-content-type-options: nosniff
x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
x-xss-protection: 1; mode=block
< atl-traceid: b04b327f99257c4d
atl-traceid: b04b327f99257c4d
< report-to: {"endpoints": [{"url": "https://dz8aopenkvv6s.cloudfront.net"}], "group": "endpoint-1", "include_subdomains": true, "max_age": 600}
report-to: {"endpoints": [{"url": "https://dz8aopenkvv6s.cloudfront.net"}], "group": "endpoint-1", "include_subdomains": true, "max_age": 600}
< nel: {"failure_fraction": 0.001, "include_subdomains": true, "max_age": 600, "report_to": "endpoint-1"}
nel: {"failure_fraction": 0.001, "include_subdomains": true, "max_age": 600, "report_to": "endpoint-1"}
< strict-transport-security: max-age=63072000; includeSubDomains; preload
strict-transport-security: max-age=63072000; includeSubDomains; preload

curl 8.1.2 (x86_64-apple-darwin22.0) libcurl/8.1.2 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.11 nghttp2/1.51.0
Release-Date: 2023-05-30
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HSTS HTTP2 HTTPS-proxy IPv6 Kerberos Largefile libz MultiSSL NTLM NTLM_WB SPNEGO SSL threadsafe UnixSockets

(I've anonymized the appropriate stuffs. And confirmed that the base64 encoded Basic Auth is identical to what -u is sending.)

[ahungry]

I notice an XSRF cookie value in the Cookie header in your manual sample - what happens in your manual test cases if you omit the cookie jar?

What happens on the Emacs side if you clear or delete this cookie jar file and then try?

Sorry that most the suggestions amount to guess and check - this is not something I am able to setup a reproduction for - if you can figure out the issue, it'd be a nice fix to incorporate/solve though.

[opie4624]

Leaving the cookie out doesn't seem to make a difference. If I nuke the cookie jar, I get a new xsrf cookie set. I've resorted to digging into request.el to see if I can make it use the -u parameter instead of doing the Base64 ourselves.

The main difference seems to be the ordering of the headers, and the only way I have managed to get the Authorization header into a place where it works, is with -u.

[ahungry]

(Ignore the last comment, I made an error on the testing :sob:)

[eval-on-point]

I had the same issue detailed here. Regenerating the API token fixed it for me.

[ajft]

Same issue, we use 2FA and cloud JIRA. I have generated an API key. I have no idea where to enter this API key. "The first time you try and connect to jira you will be asked for your username and password" no, it did not, not on the first time nor on any subsequent time. All I get are error messages in the MESSAGES buffer (that tell me to look in the MESSAGES buffer)

[ag91]

I just tried to set (setq jiralib-user "my-user-email-address") and it worked, maybe can help others

[ag91]

and if that doesn't work

(setq jiralib-token
          (cons
           "Authorization"
           (concat "Basic " (base64-encode-string (concat "<email-address>:" "<api token>") 'no-line-break))))