search

GLEIF-IT / reg-pilot

A project to manage reg-pilot related issues
Apache License 2.0
2 stars 3 forks source link

File Upload Credential/Headers issues #45

Closed bogtieba closed 3 months ago

bogtieba commented 3 months ago 
reg-pilot-api-1  | Processing signed header verification request <starlette.requests.Request object at 0x7f44d036f2c0>
reg-pilot-api-1  | processing header req <starlette.requests.Request object at 0x7f44d036f2c0>
reg-pilot-api-1  | verification input aid=EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw ser="@method": POST
reg-pilot-api-1  | "@path": /upload/EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw/e1bce01400cebb494ae68ec295e3faed859132271522486499fc801ebf2a8818
reg-pilot-api-1  | "signify-resource": EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw
reg-pilot-api-1  | "signify-timestamp": 2024-08-08T07:08:03.967000+00:00
reg-pilot-api-1  | "@signature-params: (@method @path signify-resource signify-timestamp);created=1723100883;keyid=BHerQd_5W7xwEf7_3hN7xFhh3xtjEmPdOlI5zunAt2cb;alg=ed25519" cig=0BCjl6iuBYg62Rkg7tdTDnDNbye0VRGj_of_mhQTbUCpGwI04bwIH5e77g7MbT1GKEfo3ZJLpgm5Hi0an0jPD2sC
reg-pilot-api-1  | Verify header sig started aid = EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw, cig = 0BCjl6iuBYg62Rkg7tdTDnDNbye0VRGj_of_mhQTbUCpGwI04bwIH5e77g7MbT1GKEfo3ZJLpgm5Hi0an0jPD2sC, ser = "@method": POST
reg-pilot-api-1  | "@path": /upload/EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw/e1bce01400cebb494ae68ec295e3faed859132271522486499fc801ebf2a8818
reg-pilot-api-1  | "signify-resource": EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw
reg-pilot-api-1  | "signify-timestamp": 2024-08-08T07:08:03.967000+00:00
reg-pilot-api-1  | "@signature-params: (@method @path signify-resource signify-timestamp);created=1723100883;keyid=BHerQd_5W7xwEf7_3hN7xFhh3xtjEmPdOlI5zunAt2cb;alg=ed25519"....
reg-pilot-api-1  | posting to http://vlei-verifier:7676/request/verify/EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw
vlei-verifier-1  | keri: Parsed Request:
vlei-verifier-1  | POST /request/verify/EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw {(1, 1)
vlei-verifier-1  | Hict([('Host', 'vlei-verifier:7676'), ('User-Agent', 'python-requests/2.32.3'), ('Accept-Encoding', 'gzip, deflate'), ('Accept', '*/*'), ('Connection', 'keep-alive'), ('Content-Length', '0')])
vlei-verifier-1  | bytearray(b'')
vlei-verifier-1  |
reg-pilot-api-1  | Verify sig response {"msg": "unknown EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw used to sign header"}
reg-pilot-api-1  | Upload: Exception: 404: {'msg': 'unknown EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw used to sign header'}
reg-pilot-api-1  | INFO:     172.18.0.1:50938 - "POST /upload/EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw/e1bce01400cebb494ae68ec295e3faed859132271522486499fc801ebf2a8818 HTTP/1.1" 404 Not Found

reg-pilot-api-1  | Processing signed header verification request <starlette.requests.Request object at 0x7f44d036f2c0>
reg-pilot-api-1  | processing header req <starlette.requests.Request object at 0x7f44d036f2c0>
reg-pilot-api-1  | verification input aid=EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw ser="@method": POST
reg-pilot-api-1  | "@path": /upload/EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw/e1bce01400cebb494ae68ec295e3faed859132271522486499fc801ebf2a8818
reg-pilot-api-1  | "signify-resource": EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw
reg-pilot-api-1  | "signify-timestamp": 2024-08-08T07:08:03.967000+00:00
reg-pilot-api-1  | "@signature-params: (@method @path signify-resource signify-timestamp);created=1723100883;keyid=BHerQd_5W7xwEf7_3hN7xFhh3xtjEmPdOlI5zunAt2cb;alg=ed25519" cig=0BCjl6iuBYg62Rkg7tdTDnDNbye0VRGj_of_mhQTbUCpGwI04bwIH5e77g7MbT1GKEfo3ZJLpgm5Hi0an0jPD2sC
reg-pilot-api-1  | Verify header sig started aid = EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw, cig = 0BCjl6iuBYg62Rkg7tdTDnDNbye0VRGj_of_mhQTbUCpGwI04bwIH5e77g7MbT1GKEfo3ZJLpgm5Hi0an0jPD2sC, ser = "@method": POST
reg-pilot-api-1  | "@path": /upload/EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw/e1bce01400cebb494ae68ec295e3faed859132271522486499fc801ebf2a8818
reg-pilot-api-1  | "signify-resource": EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw
reg-pilot-api-1  | "signify-timestamp": 2024-08-08T07:08:03.967000+00:00
reg-pilot-api-1  | "@signature-params: (@method @path signify-resource signify-timestamp);created=1723100883;keyid=BHerQd_5W7xwEf7_3hN7xFhh3xtjEmPdOlI5zunAt2cb;alg=ed25519"....
reg-pilot-api-1  | posting to http://vlei-verifier:7676/request/verify/EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw
vlei-verifier-1  | keri: Parsed Request:
vlei-verifier-1  | POST /request/verify/EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw {(1, 1)
vlei-verifier-1  | Hict([('Host', 'vlei-verifier:7676'), ('User-Agent', 'python-requests/2.32.3'), ('Accept-Encoding', 'gzip, deflate'), ('Accept', '*/*'), ('Connection', 'keep-alive'), ('Content-Length', '0')])
vlei-verifier-1  | bytearray(b'')
vlei-verifier-1  |
reg-pilot-api-1  | Verify sig response {"msg": "unknown EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw used to sign header"}
reg-pilot-api-1  | Upload: Exception: 404: {'msg': 'unknown EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw used to sign header'}
reg-pilot-api-1  | INFO:     172.18.0.1:50938 - "POST /upload/EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw/e1bce01400cebb494ae68ec295e3faed859132271522486499fc801ebf2a8818 HTTP/1.1" 404 Not Found

Upload: Exception: 404: {'msg': 'unknown EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw used to sign header'}

lance.byrd (Guest)morning. Yesterday we fetched the latest version of vlei and api, we built it and we tried again the upload step. The result is the same as before, 404 unknown EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw used to sign header.

Is something we do wrong on our side? I know that couple of weeks ago, Daniel Lenksjö (Guest) mentioned the register endpoint, we tried that PUT presentations/EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw with the cesr as body but got "msg": "credential EFE8-Km32lJzOa51K3IWMcctJCX8Ifu5f4BaUWdfgSbw from body of request was not found

fyi: Tiberiu Covaci

reg-pilot-api-1  | Processing signed header verification request <starlette.requests.Request object at 0x7f44d036f2c0>reg-pilot-api-1  | processing header req <starlette.requests.Request object at 0x7f44d036f2c0>reg-pilot-api-1  | verification input aid=EFE8-Km32lJzOa51K3IWM…

Hi Bogdan, thank you for updating the services and reporting this. I traced through the call stack and those messages about an "unknown used to sign header" during header verification indicates the verifier hasn't accepted your ECR credential from the login step. Recall that the three calls that require signed headers are GET /status/, GET /upload//, and POST /upload// so signed headers issues aren't seen until you execute one of those. Could you execute the GET /login step and send the logs for that step?

My initial thoughts to help integrators with identifying login and header verification problems is that perhaps we can improve the login response to better point out the login failure. And also to more easily diagnose header verification problems we could provide an endpoint that only does header verification (after login) so that an integrator can attempt a GET with signed headers (maybe a GET /login//status with signed headers) so that you can confirm login status prior to upload, check_upload, or status. However, off the top of my head the /status/ endpoint would be the simplest test currently of your signed headers since it is a GET and only requires the and the signed headers.

Also, are you able to access GITHUB reg-pilot project? I would love to have these issues posted in the reg-pilot so that your issues, my responses, etc. are seen and processed systematically with issues, comments, and PRs so that we can effectively track the work and more of the team can see the struggles/needs/questions and can respond as quickly as possible. It would also assist future integrators.

If you are okay with processing this on Github, i would be happy to take all the information you have provided and my response and put it in an issue so we can go through the workflow/fix together and then it will be easier for you to process issues/fixes with the whole team as well as see the status of what issues are being worked and we can prioritise them according to your needs, tc.

Details from an internal conversation with @2byrds

bogtieba commented 3 months ago

If this is a problem of headers, then why the reg-pilot-api is is not rejecting the request with headers error?

The error is thrown by the verifier, so why the verifier do not accept the headers while the api accepts them? Don't these 2 services use the same verification mechanism for headers?

2byrds commented 3 months ago

If this is a problem of headers, then why the reg-pilot-api is is not rejecting the request with headers error?

The error is thrown by the verifier, so why the verifier do not accept the headers while the api accepts them? Don't these 2 services use the same verification mechanism for headers?

Thank you for posting here @bogtieba ! Thats a good question.

2byrds commented 3 months ago

Note that we can simulate your same login using this test environment setup: https://github.com/GLEIF-IT/reg-pilot/blob/main/signify-ts-test/test/utils/resolve-env.ts#L148 And this integration test: https://github.com/GLEIF-IT/reg-pilot/blob/main/signify-ts-test/test/vlei-verification.test.ts#L133

2byrds commented 3 months ago

Waiting for @bogtieba to test the changes/command provided https://github.com/GLEIF-IT/reg-pilot/pull/50#discussion_r1722142514

bogtieba commented 3 months ago

Hey @2byrds ! I was able to manually run the tests I was looking for using SIGNIFY_SECRETS="A7DKYPya4oi6uDnvBmjjp" TEST_ENVIRONMENT="nordlei_demo" ROLE_NAME="unicredit-datasubmitter" REG_PILOT_API="https://reg-api-dev.rootsid.cloud" npx jest ./vlei-verification.test.ts -t "reg-pilot-api"

Currently I'm adjusting my implementation to confirm that the flow is working within our application. Hope I have a answer by tomorrow.

2byrds commented 3 months ago

Hey @2byrds ! I was able to manually run the tests I was looking for using SIGNIFY_SECRETS="A7DKYPya4oi6uDnvBmjjp" TEST_ENVIRONMENT="nordlei_demo" ROLE_NAME="unicredit-datasubmitter" REG_PILOT_API="https://reg-api-dev.rootsid.cloud" npx jest ./vlei-verification.test.ts -t "reg-pilot-api"

Currently I'm adjusting my implementation to confirm that the flow is working within our application. Hope I have a answer by tomorrow.

Thank you for the update @bogtieba . Sounds good and happy to help if you run into anything. We will be pushing out some new updates soon (in a day or so) with the SHA-256 digest validation per https://github.com/GLEIF-IT/reg-pilot/discussions/41

bogtieba commented 3 months ago

We are now able to send files to the verifier using signed requests. I used the test files as examples.
We had 2 problems

  1. We used a different aid (not the same as the signer)
  2. On the POST request, I was sending a application/zip with the body as the zip binary. The right way is to send an POST request and file field should be the body of the zip file.

We can close this now and I will raise a new one for point 1.

2byrds commented 3 months ago

We are now able to send files to the verifier using signed requests. I used the test files as examples. We had 2 problems

  1. We used a different aid (not the same as the signer)
  2. On the POST request, I was sending a application/zip with the body as the zip binary. The right way is to send an POST request and file field should be the body of the zip file.

We can close this now and I will raise a new one for point 1.

Thats great @bogtieba lets continue to improve it for easy use for you and others.