Require less scope than manage_project

[ohgo]

Hi,

The sync seems to need _manageproject scope to run. Surely it should be able to run even using api credentials with lower privilege? For instance when syncing Products, it should be enough with view:products on the source project and manage:products on the destination project. So that we don't create api clients with unnecessarily permissive privileges.

To reproduce:

docker run \
  -e SOURCE_PROJECT_KEY=<SOURCE_PROJECT_KEY> \
  -e SOURCE_CLIENT_ID=<SOURCE_CLIENT_ID> \
  -e SOURCE_CLIENT_SECRET=<SOURCE_CLIENT_SECRET> \
  -e SOURCE_AUTH_URL=<SOURCE_AUTH_URL> \
  -e SOURCE_API_URL=<SOURCE_API_URL> \
  -e TARGET_PROJECT_KEY=<TARGET_PROJECT_KEY> \
  -e TARGET_CLIENT_ID=<TARGET_CLIENT_ID> \
  -e TARGET_CLIENT_SECRET=<TARGET_CLIENT_SECRET> \
  -e TARGET_AUTH_URL=<TARGET_AUTH_URL> \
  -e TARGET_API_URL=<TARGET_API_URL> \
  commercetools/commercetools-project-sync:4.0.3 -s products

The error:

{
  "timestamp": "2021-06-30T13:07:40.184Z",
  "severity": "INFO",
  "loggerName": "com.commercetools.project.sync.product.ProductSyncer",
  "message": "Starting ProductSync from CTP project with key 'source-project' to project with key 'dest-project'"
}

{
  "timestamp": "2021-06-30T13:07:40.870Z",
  "severity": "ERROR",
  "loggerName": "sphere.oauth",
  "message": "Can't fetch tokens.",
  "stackHash": "5c696068",
  "stackTrace": "i.s.s.c.UnauthorizedException: detailMessage: io.sphere.sdk.http.HttpResponseImpl@2a75603e[statusCode=400,headers={date=[Wed, 30 Jun 2021 13:07:40 GMT], content-length=[187], server=[istio-envoy], x-envoy-upstream-service-time=[17], Alt-Svc=[clear], access-control-allow-headers=[Accept, Authorization, Content-Type, Origin, User-Agent, X-Correlation-ID], x-correlation-id=[source-project/d39b6593-751e-42a4-8c17-727c12233c0d], access-control-allow-methods=[GET, POST, DELETE, OPTIONS], pragma=[no-cache], Via=[1.1 google], access-control-allow-origin=[*], access-control-max-age=[299], content-type=[application/json; charset=utf-8], server-timing=[auth;dur=16], cache-control=[no-store]},associatedRequest=HttpRequestImpl[body=FormUrlEncodedHttpRequestBody[data=[NameValuePairImpl[name=grant_type,value=client_credentials], NameValuePairImpl[name=scope,value=manage_project:source-project]]],headers={Authorization=[**removed from output**], User-Agent=[commercetools-jvm-sdk/1.63.0 (AHC/2.1) Java/11.0.11+9 (Linux; amd64) commercetools-sync-java/5.1.2], X-Correlation-ID=[source-project/d39b6593-751e-42a4-8c17-727c12233c0d], Content-Type=[application/x-www-form-urlencoded]},httpMethod=POST,url=https://auth.europe-west1.gcp.commercetools.com/oauth/token],textInterpretedBody={\"statusCode\":400,\"message\":\"Permissions exceeded\",\"errors\":[{\"code\":\"invalid_scope\",\"message\":\"Permissions exceeded\"}],\"error\":\"invalid_scope\",\"error_description\":\"Permissions exceeded\"}]\nhttp response: <unknown>\nSDK: 1.63.0\nproject: <unknown>\nJava: 11.0.11\ncwd: /app\nsphere request: <unknown>\nadditional notes: []\nJavadoc: https://commercetools.github.io/commercetools-jvm-sdk/apidocs/io/sphere/sdk/client/UnauthorizedException.html\n\n\tat i.s.s.c.TokensSupplierImpl.parseResponse(TokensSupplierImpl.java:124)\n\t... 16 common frames omitted\nWrapped by: i.s.s.c.InvalidScopeException: detailMessage: Invalid scope error\nsummary: POST https://auth.europe-west1.gcp.commercetools.com/oauth/token failed  with response code 400 with X-Correlation-I...\n"
}

[ahmetoz]

Hi @ohgo

Could you try to pass SCOPES environment variables: -e SOURCE_SCOPES=manage_products or e TARGET_SCOPES=manage_products ? you might pass multiple values with comma: manage_customers,manage_products.

[ohgo]

Hi @ahmetoz, thanks for the tip! The app seems understand SOURCE_SCOPES and TARGET_SCOPES. And I saw the PR too, thanks for quickly fixing that. Could I suggest documenting even the minimum required scope for each resource sync (if you don't think that's obvious).

During my trial and errors, I ran into intermittent warning when syncing Products (-s products). This sounds unrelated to this scope issue, in which case we can close this anyway. But I'd like your confirmation, and perhaps a pointer on how to troubleshoot if possible.

{
  "timestamp": "2021-07-01T13:08:01.857Z",
  "severity": "INFO",
  "loggerName": "com.commercetools.project.sync.product.ProductSyncer",
  "message": "Starting ProductSync from CTP project with key 'source-project' to project with key 'target-project'"
}

Jul 01, 2021 1:08:04 PM org.javamoney.moneta.DefaultMonetaryContextFactory createMonetaryContextNonNullConfig
INFO: Using custom MathContext: precision=256, roundingMode=HALF_EVEN

{
  "timestamp": "2021-07-01T13:08:05.397Z",
  "severity": "WARN",
  "loggerName": "com.commercetools.project.sync.product.ProductSyncer",
  "message": "java.lang.IllegalArgumentException: argument \"content\" is null",
  "stackHash": "1adc93c3",
  "stackTrace": "j.l.IllegalArgumentException: argument \"content\" is null\n\tat c.f.j.d.ObjectReader._assertNotNull(ObjectReader.java:2366)\n\tat c.f.j.d.ObjectReader.readValue(ObjectReader.java:1665)\n\tat i.s.s.j.SphereJsonUtils.lambda$readObject$9(SphereJsonUtils.java:238)\n\tat i.s.s.j.SphereJsonUtils.executing(SphereJsonUtils.java:324)\n\tat i.s.s.j.SphereJsonUtils.readObject(SphereJsonUtils.java:238)\n\tat c.c.s.c.h.GraphQlBaseRequestImpl.deserializeWithResourceName(GraphQlBaseRequestImpl.java:77)\n\tat c.c.s.c.m.ResourceIdsGraphQlRequest.deserialize(ResourceIdsGraphQlRequest.java:27)\n\tat c.c.s.c.m.ResourceIdsGraphQlRequest.deserialize(ResourceIdsGraphQlRequest.java:14)\n\tat i.s.s.c.SphereClientImpl.parse(SphereClientImpl.java:139)\n\tat i.s.s.c.SphereClientImpl.processHttpResponse(SphereClientImpl.java:129)\n\tat i.s.s.c.SphereClientImpl.lambda$executeWithRecover$4(SphereClientImpl.java:86)\n\tat j.u.c.CompletableFuture$UniApply.tryFire(Unknown Source)\n\tat j.u.c.CompletableFuture$Completion.exec(Unknown Source)\n\tat j.u.c.ForkJoinTask.doExec(Unknown Source)\n\tat j.u.c.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)\n\tat j.u.c.ForkJoinPool.scan(Unknown Source)\n\tat j.u.c.ForkJoinPool.runWorker(Unknown Source)\n\tat j.u.c.ForkJoinWorkerThread.run(Unknown Source)\n"
}

{
  "timestamp": "2021-07-01T13:08:05.465Z",
  "severity": "INFO",
  "loggerName": "com.commercetools.project.sync.product.ProductSyncer",
  "message": "Summary: 0 product(s) were processed in total (0 created, 0 updated, 0 failed to sync and 0 product(s) with missing reference(s)).",
  "statistics": {
    "updated": 0,
    "created": 0,
    "failed": 0,
    "processed": 0,
    "latestBatchProcessingTimeInDays": 0,
    "latestBatchProcessingTimeInHours": 0,
    "latestBatchProcessingTimeInMinutes": 0,
    "latestBatchProcessingTimeInSeconds": 0,
    "latestBatchProcessingTimeInMillis": 1,
    "latestBatchHumanReadableProcessingTime": "0d, 0h, 0m, 0s, 1ms",
    "reportMessage": "Summary: 0 product(s) were processed in total (0 created, 0 updated, 0 failed to sync and 0 product(s) with missing reference(s)).",
    "numberOfProductsWithMissingParents": 0
  }
}

[ahmetoz]

Could I suggest documenting even the minimum required scope for each resource sync (if you don't think that's obvious).

I would like to document that but it depends on the what are you trying to do with app as it might have many combinations, so scope of each combination might be different. So I would suggest to check https://docs.commercetools.com/api/scopes page and try to find minimum scope needed for your case.

For the case that you have shared, I have not seen this issue before, the stack trace also does not say much, it looks like the issue happened on parsing the graphql response it might be the query requires more scope, as this normally not happens. The result also interesting as the nothings seems like failed 🤔

[ohgo]

I would like to document that but it depends on the what are you trying to do with app as it might have many combinations,

You're right. I think it's not that hard to figure out anyway.

Yeah, I'll see if I can nail down what is causing this intermittent warning later. Feel free to close this issue whenever you feel appropriate. Thanks again for the help!

[ritammv]

@ohgo did you manage to find a solution for this? Having similar issues

[heshamMassoud]

@ritammv I've hit this problem a couple of days ago, and it was because of not enough scope in my API client. So try giving your client a bigger scope.

[butenkor]

Hi @heshamMassoud thanks for heads up :) The confusion might be here due to custom objects which have quite extensive permission requirements.

[heshamMassoud]

@butenkor From the stacktrace posted above (and also similar to the one I got), seems something related to the GraphQL requests no? :)

j.l.IllegalArgumentException: argument \"content\" is null\n\tat c.f.j.d.ObjectReader._assertNotNull(ObjectReader.java:2366)\n\tat c.f.j.d.ObjectReader.readValue(ObjectReader.java:1665)\n\tat i.s.s.j.SphereJsonUtils.lambda$readObject$9(SphereJsonUtils.java:238)\n\tat i.s.s.j.SphereJsonUtils.executing(SphereJsonUtils.java:324)\n\tat i.s.s.j.SphereJsonUtils.readObject(SphereJsonUtils.java:238)\n\tat c.c.s.c.h.GraphQlBaseRequestImpl.deserializeWithResourceName(GraphQlBaseRequestImpl.java:77)\n\tat c.c.s.c.m.ResourceIdsGraphQlRequest.deserialize(ResourceIdsGraphQlRequest.java:27)\n\tat c.c.s.c.m.ResourceIdsGraphQlRequest.deserialize(ResourceIdsGraphQlRequest.java:14)\n\tat i.s.s.c.SphereClientImpl.parse(SphereClientImpl.java:139)\n\tat i.s.s.c.SphereClientImpl.processHttpResponse(SphereClientImpl.java:129)\n\tat i.s.s.c.SphereClientImpl.lambda$executeWithRecover$4(SphereClientImpl.java:86)\n\tat j.u.c.CompletableFuture$UniApply.tryFire(Unknown Source)\n\tat j.u.c.CompletableFuture$Completion.exec(Unknown Source)\n\tat j.u.c.ForkJoinTask.doExec(Unknown Source)\n\tat j.u.c.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)\n\tat j.u.c.ForkJoinPool.scan(Unknown Source)\n\tat j.u.c.ForkJoinPool.runWorker(Unknown Source)\n\tat j.u.c.ForkJoinWorkerThread.run(Unknown Source)\n

[ritammv]

I had the same error as original poster and by increasing the scope it worked for me! if syncing custom objects you need the view_customers scope on the source (hope that helps)

On Fri, 23 Jun 2023 at 13:07, Hesham Massoud @.***> wrote:

@butenkor https://github.com/butenkor From the stacktrace posted above (and also similar to the one I got), seems something related to the GraphQL requests no? :)

j.l.IllegalArgumentException: argument \"content\" is null\n\tat c.f.j.d.ObjectReader._assertNotNull(ObjectReader.java:2366)\n\tat c.f.j.d.ObjectReader.readValue(ObjectReader.java:1665)\n\tat i.s.s.j.SphereJsonUtils.lambda$readObject$9(SphereJsonUtils.java:238)\n\tat i.s.s.j.SphereJsonUtils.executing(SphereJsonUtils.java:324)\n\tat i.s.s.j.SphereJsonUtils.readObject(SphereJsonUtils.java:238)\n\tat c.c.s.c.h.GraphQlBaseRequestImpl.deserializeWithResourceName(GraphQlBaseRequestImpl.java:77)\n\tat c.c.s.c.m.ResourceIdsGraphQlRequest.deserialize(ResourceIdsGraphQlRequest.java:27)\n\tat c.c.s.c.m.ResourceIdsGraphQlRequest.deserialize(ResourceIdsGraphQlRequest.java:14)\n\tat i.s.s.c.SphereClientImpl.parse(SphereClientImpl.java:139)\n\tat i.s.s.c.SphereClientImpl.processHttpResponse(SphereClientImpl.java:129)\n\tat i.s.s.c.SphereClientImpl.lambda$executeWithRecover$4(SphereClientImpl.java:86)\n\tat j.u.c.CompletableFuture$UniApply.tryFire(Unknown Source)\n\tat j.u.c.CompletableFuture$Completion.exec(Unknown Source)\n\tat j.u.c.ForkJoinTask.doExec(Unknown Source)\n\tat j.u.c.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)\n\tat j.u.c.ForkJoinPool.scan(Unknown Source)\n\tat j.u.c.ForkJoinPool.runWorker(Unknown Source)\n\tat j.u.c.ForkJoinWorkerThread.run(Unknown Source)\n

— Reply to this email directly, view it on GitHub https://github.com/commercetools/commercetools-project-sync/issues/304#issuecomment-1604189758, or unsubscribe https://github.com/notifications/unsubscribe-auth/AP4KV5ZTK3IQWMEUD5ZQGHTXMWBHFANCNFSM47SLYQ3A . You are receiving this because you were mentioned.Message ID: @.*** com>