DP-3T / dp3t-sdk-backend

The backend implementation for DP3T
Mozilla Public License 2.0
199 stars 87 forks source link

API in Swagger Demo is Not working - always returns 400 #212

Closed vijayakumarmariyappan closed 4 years ago

vijayakumarmariyappan commented 4 years ago

@ubamrein @martinalig

Thank you so much for your contribution to this project. You have provided demo for Android and iOS. That is good. But for Backend project found no clear steps to run. I'm sorry to say this.

I'm from Node JS platform. I can understand Java code as well. Somehow I did run the backend server by using the existing documentation. It runs. If I access the below API 1, then it responded with "Hello from DP3T WS". But for all other APIs, it returns 400 as status code. according to the documentation It says "Invalid base64 encoding in expose request". I really tried various input data inside body, but it always returns the same. Then I was thinking it has an issue in my jar file. Then I loaded "sdk.yaml" file in the path "/home/vijay/Desktop/dp3t-sdk-backend/documentation/yaml" in https://editor.swagger.io/.

I tried the below on that swagger editor, found that same behaviour as my local jar.

API 1: Request: GET http://localhost:8080/v1 OR https://demo.dpppt.org/v1 Response: Hello from DP3T WS

API 2: Request: POST http://localhost:8080/v1/exposed OR https://demo.dpppt.org/v1/exposed Body: { "key": "dGVzdA==", "keyDate": 1596672000, }

Here dGVzdA== is base64 encoded string of word "test". 1596672000 - timestamp is equivalent to 08/06/2020 @ 12:00am (UTC)- converted by https://www.unixtimestamp.com/index.php

Response: 400

I would like to know how can I use the backend app properly. is the below correct? First I'm storing the exposee's unique id(Example:email-id) in DB, by using this API(https://demo.dpppt.org/v1/exposed) Second I can get the list of friend who may have affected by his contact, by using https://demo.dpppt.org/v1/gaen/exposed/:timestamp

Did i do any mistakes on sending the data in the POST body? Kindly please help us regarding this. If you could add demo for backend, really really helpful to everyone. Thanks a lot :+1:

ubamrein commented 4 years ago

Hi!

The doumentation is a bit unprecise. I will update it accordingly.

Before the controller is executed, the @Valid annotations makes sure that the model is valid. The json you are providing is not valid, hence springboot rejects it with a 400 BadRequest.

The various validations are needed, since we want to guarantee a constant payload size for every request (so fake and real requests can't be distinguished, hence it is not easy to gather information on possible infected people by just looking at encrypted traffic).

So in your case there are for sure two mistakes:

1) The model for exposed should be a list of keys (plus some more properties) 2) The keys need to be a base64 encoding of 16 bytes (hence 24 characters)

For further validations please take a look at the models and what @Validation annotations they have. For example GaenRequest:

public class GaenRequest {
    @NotNull
    @NotEmpty
    @Valid
    @Size(min = 14, max = 30)
    @Documentation(description = "Between 14 and 30 Temporary Exposure Keys - zero or more of them might be fake keys. Starting with EN 1.5 it is possible that clients send more than 14 keys.")
    private List<GaenKey> gaenKeys;

    @NotNull
    @Documentation(description = "Prior to version 1.5 Exposure Keys for the day of report weren't available (since they were still used throughout this day RPI=144), so the submission of the last key had to be delayed. This Unix timestamp in milliseconds specifies, which key date the last key (which will be submitted on the next day) will have. The backend then issues a JWT to allow the submission of this last key with specified key date. This should not be necessary after the Exposure Framework is able to send and handle keys with RollingPeriod < 144 (e.g. only valid until submission).")
    private Integer delayedKeyDate;

    public List<GaenKey> getGaenKeys() {
        return this.gaenKeys;
    }

    public void setGaenKeys(List<GaenKey> gaenKeys) {
        this.gaenKeys = gaenKeys;
    }

    public Integer getDelayedKeyDate() {
        return this.delayedKeyDate;
    }

    public void setDelayedKeyDate(Integer delayedKeyDate) {
        this.delayedKeyDate = delayedKeyDate;
    }
}

The models themselves should also provide a little bit more feedback to what each parameter means.

Regarding the demo, I can't quite follow what you mean.

We already provide two ways to test and give insights to how the backend is working:

1) You can use the apps and point them to your backend deployments 2) Have a look at the Test files which build the correct payloads

You are of course free to provide a PR adding some postman collections or similar. This is currently for us not high priority as we are preparing for multiple API changes coming in the future.

vijayakumarmariyappan commented 4 years ago

@ubamrein Thank you for your help :)

How I configured Swagger in https://editor.swagger.io/ visited that tool site, then uploaded this below file/copy paste the content into https://editor.swagger.io/ File: https://github.com/DP-3T/dp3t-sdk-backend/blob/develop/documentation/yaml/sdk.yaml

You had mentioned as below,

**So in your case there are for sure two mistakes:

1.The model for exposed should be a list of keys (plus some more properties)

  1. The keys need to be a base64 encoding of 16 bytes (hence 24 characters)**

1.The model for exposed should be a list of keys (plus some more properties) tried 2 API from your swagger documentation. v1/exposed v1/exposedlist

2. The keys need to be a base64 encoding of 16 bytes (hence 24 characters) I have used this below tool and got the 24 characters for key Screenshot from 2020-08-06 10-38-30

Though I referred this test case where they are using UTF 8 for encoding base 64. one more key was there "onset". I tried that also, but still 400 response I received. https://github.com/DP-3T/dp3t-sdk-backend/blob/develop/dpppt-backend-sdk/dpppt-backend-sdk-ws/src/test/resources/requests.http

Based on that I have tried with the below POST body, but still I'm getting 400.

Try 1: Screenshot from 2020-08-06 10-33-32 Try 1 Result: Screenshot from 2020-08-06 10-33-45

Try 2: Screenshot from 2020-08-06 10-31-38 Try 2 Result: Screenshot from 2020-08-06 10-32-03

What I'm trying to achieve is, working payload/POST body for your swagger documentation by using https://editor.swagger.io/. If I could see the working copy in Swagger then I can get the idea. Kindly please share the working payload for the APIs.

My Goal: Run the dp3t-sdk-backend, without modification in AWS, then use 2 APIs

  1. For exposing
  2. Retrieving list of possible contacts from the timestamp

As I'm stucked here, I unable to move forward. Kindly please help us regarding tihs. Thank you for your help. Please update me if any further details you needed from your side. Thank you :)

ubamrein commented 4 years ago

Oh so you are not trying to use the ExposureFramework?

There are two completely different controllers:

The two approaches have different KeySizes as can be seen by the different ValidationUtils they are using:

    @Value("${ws.app.key_size: 32}")
    int keySizeBytes;

        @Bean
    public ValidationUtils dpptValidationUtils() {
        return new ValidationUtils(keySizeBytes, Duration.ofDays(retentionDays), releaseBucketDuration);
    }

        @Value("${ws.app.gaen.key_size: 16}")
    int gaenKeySizeBytes;

        @Bean
    public ValidationUtils gaenValidationUtils() {
        return new ValidationUtils(gaenKeySizeBytes, Duration.ofDays(retentionDays), releaseBucketDuration);
    }

For information on what is injected take a look at the constructors of the two controllers in the WSBaseConfig:

        @Bean
    public DPPPTController dppptSDKController() {
        ValidateRequest theValidator = requestValidator;
        if (theValidator == null) {
            theValidator = new NoValidateRequest(dpptValidationUtils());
        }
        return new DPPPTController(dppptSDKDataService(), appSource, exposedListCacheControl, theValidator,
                dpptValidationUtils(), releaseBucketDuration, requestTime);
    }
        @Bean
    public GaenController gaenController() {
        ValidateRequest theValidator = gaenRequestValidator;
        if (theValidator == null) {
            theValidator = backupValidator();
        }
        return new GaenController(insertManager(),gaenDataService(), fakeKeyService(), theValidator, gaenSigner(),
                gaenValidationUtils(), Duration.ofMillis(releaseBucketDuration), Duration.ofMillis(requestTime),
                Duration.ofMillis(exposedListCacheControl), keyVault.get("nextDayJWT").getPrivate());
    }

For the DPPPTController here is a working CURL request:

POST

curl -vvv -X 'POST' --data '{"key": "MDEyMzQ1Njc4OWFiY2RlZjAxMjM0NTY3ODlhYmNkZWY=","keyDate": "1596693451000","fake": "0"}' -H 'Content-Type: application/json'  http://localhost:8080/v1/exposed

GET

curl -vvv http://localhost:8080/v1/exposed/1596499200000 

Btw. if you use Postman Collections it is easier to prepare data in JS. You can import OpenAPI 3 descriptions as well.

vijayakumarmariyappan commented 4 years ago

@ubamrein

Thank you for your quick reply.

you are not trying to use the ExposureFramework? I want to use the Apple & Google's ExposureFramework.

So you meant to say I have to use /v1/gaen/* Could you please give example for those APIs.

Thanks in advance :+1:

ubamrein commented 4 years ago

For the Exposure Notification there are multiple different validations going on. It is hard to just give a payload. As I said, the easiest is to use the android app set the URL to your custom backend an intercept the call with a web proxy.

This curl works on the feature/insert-manager branch (for now)

curl -vvv -H 'Content-Type: application/json'  -H 'Accept: */*' -H 'User-Agent: org.dpppt.dp3t;1.0.8;200726.2312.218;iOS;13.6' -H 'Accept-Language: de-ch' --data-binary '{"fake":1,"delayedKeyDate":2661120,"gaenKeys":[{"rollingPeriod":144,"rollingStartNumber":2659536,"transmissionRiskLevel":0,"keyData":"TfBWF24K+jDu4JyBNsYccA==","fake":1},{"rollingPeriod":144,"rollingStartNumber":2659392,"transmissionRiskLevel":0,"keyData":"Uoq\/mwjaU+MZlszweBiLkw==","fake":1},{"rollingPeriod":144,"rollingStartNumber":2659248,"transmissionRiskLevel":0,"keyData":"I6xK64SRRrp+zXMIFYfH8w==","fake":1},{"rollingPeriod":144,"rollingStartNumber":2659104,"transmissionRiskLevel":0,"keyData":"rxPWEr+IlThcU6ZOZPg6UA==","fake":1},{"rollingPeriod":144,"rollingStartNumber":2658960,"transmissionRiskLevel":0,"keyData":"AcudT9l7ccZdQ2jOjbHb9g==","fake":1},{"rollingPeriod":144,"rollingStartNumber":2658816,"transmissionRiskLevel":0,"keyData":"peekB9dvtwebHqSN2UXLgw==","fake":1},{"rollingPeriod":144,"rollingStartNumber":2658672,"transmissionRiskLevel":0,"keyData":"sl7gQUcrxT5LtR6zp+Pg9Q==","fake":1},{"rollingPeriod":144,"rollingStartNumber":2658528,"transmissionRiskLevel":0,"keyData":"Gp8gqWdHvwo4YH3Dc2QVMA==","fake":1},{"rollingPeriod":144,"rollingStartNumber":2658384,"transmissionRiskLevel":0,"keyData":"eWkACtPAD8lF90hZh7BZiw==","fake":1},{"rollingPeriod":144,"rollingStartNumber":2658240,"transmissionRiskLevel":0,"keyData":"Lz87g8bXYs61ZLXg+g5ASQ==","fake":1},{"rollingPeriod":144,"rollingStartNumber":2658096,"transmissionRiskLevel":0,"keyData":"PGCQ1XB8lY\/QTT9cfu\/5kQ==","fake":1},{"rollingPeriod":144,"rollingStartNumber":2657952,"transmissionRiskLevel":0,"keyData":"hUaefAOfgJrEKtBtInI1bA==","fake":1},{"rollingPeriod":144,"rollingStartNumber":2657808,"transmissionRiskLevel":0,"keyData":"qLQQ+6oxfsQhuTxC1jz0aQ==","fake":1},{"rollingPeriod":144,"rollingStartNumber":2657664,"transmissionRiskLevel":0,"keyData":"ntGdHxyPGUPgkgoWPNkLHA==","fake":1}]}' --compressed 'http://localhost:8080/v1/gaen/exposed'
vijayakumarmariyappan commented 4 years ago

@ubamrein Thank you so much for your help :) So kind of you.

The shared curl works. I need to explain all the pieces to the Mobile team about the payload and each key & data details. That's why I requested you. After completing this exposed API, I'm trying to get the connection details by using the below, but returns as 404. did I do any mistakes?

curl --location --request GET 'http://localhost:8080/v1/gaen/exposed/2661120'

Thank you :+1:

ubamrein commented 4 years ago

You should look into the newest swagger file, as everything is explained in it.

If you look at the possible responses you see for 404:

invalid starting key date, doesn't point to midnight UTC- publishedAfter is not at the beginning of a batch release time, currently 2h

Further, as the API documentation states, your date should be in ms since UNIX Epoch and at 00:00 UTC:

Requested date for Exposed Keys retrieval, in milliseconds since Unix epoch (1970-01-01). It must indicate the beginning of a TEKRollingPeriod, currently midnight UTC.

Also, if you look at the models in the API Documentation, you'll find examples and desccriptions on what each parameter means and validations are needed.

vijayakumarmariyappan commented 4 years ago

@ubamrein Thank you for your help :)

In your previously shared CURL, I saw that delayedKeyDate as 2661120(01/31/1970 @ 7:12pm (UTC)), that was posted to server in v1/gaen/exposed API. I did the below changes in the given POST data

updated all "fake": 1 into "fake": 0, so it will not skip the key while saving. then with those data I tried in localhost, it returned 200 status code. So we exposed the list of keys.

Screenshot from 2020-08-06 14-05-29

So here to use GET method for the same, 2661120 should be given here. isn't it? I tried with publishedafter=1 & publishedafter=0. But for both I'm getting 404.

Screenshot from 2020-08-06 14-14-21

I'm using this tool to convert to timestamp. https://www.unixtimestamp.com/index.php

Could you please share the curl for GET method http://localhost:8080/v1/gaen/exposed/:timestamp?publishedafter=0 against your POST curl Example with fake=0

Please review the screenshots, update me if you need more details.

Thank you for your time :)

ubamrein commented 4 years ago

No, not at all..

Also see (taken from README):

GET Returns a list of keys, which were used at timestamp. Note that timestamp needs to be epoch milliseconds. Since the behaviour of Android and iOS aren't the same, the optional publishedAfter parameter is added. If set only keys, which were received after publishAfter are returned. This request returns ZIP file containing export.bin and export.sig, where the keys and the signature are stored, as need by the EN framework. The class for signing and serializing is ProtoSignature.

Also note that release of keys is delayed, and aligned with the bucket boundaries (defaults to 2h).

Buckets only exist for 14 days, so everything older than 14 days will always return 404

vijayakumarmariyappan commented 4 years ago

@ubamrein Thank you so much for answering & writing effort that you put here. I will communicate with mobile team regarding this.

So far what I understood is,

  1. Deploy this dp3t-sdk-backend app in remote.
  2. Android and iOS app will insert/update the dp3t-sdk-backend's DB.
  3. while sending the "keyData" to that dp3t-sdk-backend server, inside the "keyData" we will hide user specific details there.
  4. When the person got corona positive, then when he reports to my application server(node server) through mobile app, the mobile app will send the list of keysData(contactees) to the server.
  5. The application server can send the notification or something with that.

If you have any better ideas, then you could share here. Thank you!

ubamrein commented 4 years ago

I will stop discussion here and close the issue, as the API is working as expected.

How the app and the backend is working is documented in this resp. the app repositories. If you want further inspirations check out the reference implementations by Google and Apple.