Bysmyyr commented 4 years ago

Postman API PUT to collections endpoint gives 500 and 504 and sometimes returns an empty body. We update the collections to 25 users from the ci run in wercker which runs 10 updates concurrently. Sometimes all they go through fine, sometimes there is one or two fail which is covered by retry logic but increasingly lately it fails so badly than 10 retries are not enough. Collections are quite big, about 600 requests altogether. It also generates double collections(some empty, some not).

500{"error":{"name":"serverError","message":"Something went wrong with the server."}}

504:

<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>

preethammavin commented 4 years ago

@Bysmyyr can you write to us at help@getpostman.com. Would want to take a deeper look at this, I would suggest breaking the collection down into smaller collections of size (preferably lesser than 25 MB) as I see it is timeout error there.

abiancu commented 4 years ago

Trying to make a GET request to https://api.covid19api.com/summary but getting a 504 error. Was working fine on 05/02/2020

504 Gateway Time-out

nginx/1.14.0 (Ubuntu)

Bysmyyr commented 4 years ago

@abiancu Your error is not even closely related to this. This is about postman internal API, not using poistman for some other API

mohitranka commented 3 years ago

@Bysmyyr - We have made few performance improvements to these endpoints in the last month. I'd like to know if you are still facing issues around big collection updates.

galbarm commented 3 years ago

@mohitranka I'm still facing the issue. You can see more details in support ticket #60411

Specifically, for update collection request with body around 1MB, the "Something went wrong with the server." error is returned after 1 minute.

Also note that collections API only operates on full collections. You cannot update a single request/folder within a collection. As a result, there is no workaround for this issue.

Bysmyyr commented 3 years ago

@mohitranka Thanks for your reply. Our issues have went worse in last months.

Raj-postman commented 3 years ago

Our team checked on this further after the comment @Bysmyyr - may I know if it helped?

@galbarm We haven't heard back from you on the ticket. Just checking to see if the recent improvements helped?

galbarm commented 3 years ago

Our team checked on this further after the comment @Bysmyyr - may I know if it helped?

@galbarm We haven't heard back from you on the ticket. Just checking to see if the recent improvements helped?

Hi, I can't tell if it helped because my workaround was to delete-create the whole collection for each update instead of using the update API, and this is how my system currently works. This is not perfect because a failure after delete but before the creation is dangerous, but I had no other option.

galbarm commented 3 years ago

Seems like you have some regression that causes this error to now occur for every (large enough) request. Even the delete-create workaround doesn't help now. You may also want to know that the request takes place even though an error is returned. As a result and to deal with that, this is how our code currently looks like:

        case result if result.status == 200 => result.body[JsValue]

        //workaround the slow Postman servers. see https://github.com/postmanlabs/postman-app-support/issues/8213
        case result if result.status == 500 && result.body.contains("Something went wrong with the server") =>
          logger.debug("finishing with success (even though postman 500), because we've observed that the collection gets updated regardless of the error.")
          result.body[JsValue]

        case result => throw HttpResponseException(result)

Please try not to change anything, unless you actually fix it.

RonMiltonPing commented 3 years ago

Seeing the same behavior. Our collection is about 3 MB, 300+ requests with a lot of documentation. Put call to Collections returns a 504 after a minute but the collection gets updated a few minutes later. Have not tried the delete-create approach yet.

kszafran commented 3 years ago

We had to remove most of the descriptions from our collection to lower the overall size, and now we get 500s only sometimes... It's still pretty disappointing.

AndrewGuenther commented 2 years ago

I'm seeing the 500 behavior as well. My collection file is only 350KB and I get a 500 error 100% of the time, but the collection does actually update. Similar use case to the requester: We want to update a collection from within our CI system. Here's the exact curl request:

curl -v --location -g --request PUT "https://api.getpostman.com/collections/$collection_uid" \
--header "X-Api-Key: $POSTMAN_API_KEY" \
--header 'Content-Type: application/json' \
--data-binary '@postman.json'

juliehubs commented 2 years ago

I have also been seeing this issue, has there been any solution for this? Thanks!

kszafran commented 2 years ago

I our case (same company as the OP), this is still the workaround that we're using:

We had to remove most of the descriptions from our collection to lower the overall size, and now we get 500s only sometimes... It's still pretty disappointing.

We'll be looking into alternative ways to publish our API documentation. For just making requests from the Postman app we can deal with no descriptions. I haven't seen 500s in a long time now.

AndrewGuenther commented 2 years ago

My workaround has been no longer using Postman :man_shrugging:

kszafran commented 2 years ago

@AndrewGuenther Have you switched to some alternative solution that you could recommend?

AndrewGuenther commented 2 years ago

I've been using Paw and have been pretty happy with it so far.

bpedro commented 2 years ago

Hey everyone, thanks for all your comments and patience with this issue.

We have been working on a new way of updating individual collection items (requests, folders, responses).

Please let me know if you'd be interested in testing the new endpoint.

gbarmashiahflir commented 2 years ago

Hey everyone, thanks for all your comments and patience with this issue.

We have been working on a new way of updating individual collection items (requests, folders, responses).

Please let me know if you'd be interested in testing the new endpoint.

I'll be happy to

gbarmashiahflir commented 2 years ago

@bpedro Can I get access to the new endpoint? Our API is large and is suffering from serious issues when trying to update the whole collection.

bpedro commented 2 years ago

Hi @gbarmashiahflir, thanks for your patience.

We are getting in touch with you privately with information about the individual collection items operations.

abrunke commented 2 years ago

We have recently started to get 504 errors when retrieving the collection via the post API using Newman.

Newman logs are as follows

2022-10-27T12:02:27.091-07:00   error: collection could not be loaded

2022-10-27T12:02:27.091-07:00   Error fetching the collection from the provided URL. Ensure that the URL is valid.

2022-10-27T12:02:27.091-07:00   Error fetching collection, the provided URL returned status code: 504

On retrying this is fine but it does cause the test via Newman to fail which is annoying.

Any suggestions? I can share the URL privately if that helps

bpedro commented 2 years ago

Hi @abrunke,

Thanks for your feedback. Can you please contact Postman support and share the URL privately?

https://support.postman.com/hc/en-us/requests/new/

davidespihernandez commented 1 year ago

Hi! Related to the PUT operation failing for big collections, we already have exposed an endpoint that allows updating collection individual items (requests, folders, and responses). You can find documentation and examples here: https://www.postman.com/postman/workspace/postman-public-workspace/collection/16473433-f2f0cb8f-609d-443f-913c-9831187c326e?ctx=documentation

In addition, we have been improving the performance of the PUT and POST collections endpoints, but there will always be a limitation... I mean, we won't process a collection of 100MB in a single PUT. In case of huge collections, the recommendation is to split them into smaller ones or use the collection individual items endpoint if possible.

davidespihernandez commented 1 year ago

For future reference:

The POST and PUT collection requests can fail if the payload is too big, mainly because of an internal timeout.
In case you need to manipulate big collections, you can use the collection items API.
In addition, in November 2023 new APIs that allow to copy and move folders, requests, and responses between collections (or in the same collection) will be exposed publicly.
We will continue improving the efficiency of the PUT and POST collection to be able to manage bigger payloads, but there will always be a limitation.

postmanlabs / postman-app-support

Postman api returns 500 and 504 #8213

504 Gateway Time-out