search

commercetools / commercetools-sync-java

Java library for importing and syncing (taking care of changes) data into one or more commercetools projects from external data files or from another commercetools project.
https://commercetools.github.io/commercetools-sync-java
Apache License 2.0
32 stars 37 forks source link

io.sphere.sdk.client.RequestEntityTooLargeException Error #659

Closed jixunchen closed 3 years ago

jixunchen commented 3 years ago

Describe the bug Synchronization process throws "Error 413 (Request Entity Too Large)" consistently when making custom_object query to CT. Here is how the URL looks like: (( https://api.us-central1.gcp.commercetools.com/{{projectKey}/custom-objects?where=container%3D%22commercetools-sync-java.UnresolvedReferencesService.productDrafts%22&where=key+in+%28%22f74343f371aecbe84f1f3be9f534c65ee26b4bbe%22... and 499 more...
)

To Reproduce In our server which has about 20K of products.

Step 1: Sync ProductTypes. -s productTypes Step 2: Sync Categories. -s categories Step 3: Sync Products. -s products

Expected behavior An java.util.concurrent.CompletionException: io.sphere.sdk.client.RequestEntityTooLargeException occurs.

It looks like a lot of custom objects were created from unresolved-reference-object and the predicate to query them are simply too big (about the size of 25K) to cause the "entry too big" error.

ahmetoz commented 3 years ago

Hi @jixunchen thanks for opening the issue, we took it to our backlog to implement proper chunking of unresolved custom objects.

ahmetoz commented 3 years ago

Hi @jixunchen sorry for the late reply, we already released the change on java-sync with https://github.com/commercetools/commercetools-sync-java/releases/tag/3.2.0 , but not in project-sync yet.

ahmetoz commented 3 years ago

Hi @jixunchen, My colleague @praveenkumarct released a change in project-sync, so the fix that you describe is handled on both projects, See: https://github.com/commercetools/commercetools-project-sync/releases/tag/3.11.0.

Hereby I'm closing the issue, please let us know when the issue is not resolved in your case, so we could reopen the issue again.