eBay / restcommander

Fast Parallel Async HTTP client as a Service to monitor and manage 10,000 web servers. (Java+Akka)
Apache License 2.0
902 stars 132 forks source link

Empty reply from server and maxConcurrency in Dashboard for "commandToSingleTargetServer" #17

Closed agustintorres closed 8 years ago

agustintorres commented 8 years ago

Two issues:

Issue 1: I am using the "commandToSingleTargetServer" as follows:

small request:

curl -X POST --data-binary "@myfile-small.json" http://<myrestcommanderURL>/commands/commandToSingleTargetServer
{"result":"ADHOCDATASTORE_NODEGROUP_20160920082707099-0700","time":"2016.09.20.08.27.12.601-0700"}

large request:

curl -X POST --data-binary "@myfile.json" http://<myrestcommanderURL>/commands/commandToSingleTargetServer
curl: (52) Empty reply from server

where the only difference between the two commands is that the "small request" is for only 10 URLs while "large request" is for 20,000 URLs (all going to the same server). As you can see, the small request response is a JSON with result and time, but for the large request it is an empty response. Why is this?

Further, I'm trying to figure out a way to get at the results, even for the small request. Using the "result" in the JSON, I could get to:

http://<myrestcommanderURL>/nodeGroups/nodeGroupType?nodeGroupType=ADHOCDATASTORE_NODEGROUP_20160920082707099-0700&isAdhoc=true

However, the actual results are here:

http://<myrestcommanderURL>/logs/logContent/20160920082712522-0700~ADHOCDATASTORE_NODEGROUP_20160920082707099-0700~GET_NETWORTH.jsonlog.txt

And I cannot get to them because the "20160920082712522-0700" part of the URL is a timestamp that does not match the timestamp I get in the response (which is "2016.09.20.08.27.12.601-0700").

Issue 2: Even though the command has a maxConcurrency set to 4, when I look at the Dashboard, I see the "Max Concurrency" column display 500 for my task. Why is this? I can't be sure which maxConcurrency it is actually using nor where the 500 comes from.

jeffpeiyt commented 8 years ago

hi @agustintorres , thanks for trying out REST Commander.

Current implementation does not support async job calling. You will have to wait for the response to come back. We decided to implement the async job with progress tracking in Parallec.io with many other improvements and flexibility: check example

The maxConcurrency should work for the commands. the first 0 after the /act/index.rss is the concurrency (default as 1500) , could you please try this and modified the 1st 0 and see if it works? http://www.restcommander.com/documentation.html#Commands

thanks,

GET_CRAIGSLIST_RSS GET http 80 /act/index.rss 0 0 5000 SUPERMAN_GLOBAL
```$AM_FULL_CONTENT 
agustintorres commented 8 years ago

@jeffpeiyt Thanks for the prompt reply. I actually do not want async job calling. I'm happy waiting for the response as long as it's needed. My question was why for the second curl request, I get "(52) Empty reply from server". Do you think this is most likely an issue on my end, and restcommander itself should return a response once it's done, even if it takes 10 minutes?

In addition, I'm not sure why the response that does work (for the first curl command) contains only a JSON with "result" and "time" rather than the full JSON response object with the actual responses from the server (i.e. similar to the full responses displayed at http://www.restcommander.com/usecase.html). Any idea why?

For maxConcurrency, this is my command:

GET_NETWORTH GET http 80 /networth/$UID 4 -1 -1 SUPERMAN_GLOBAL


As you can see, the maxConcurrency is set to 4 and this is fine in the Commands display:

![screen shot 2016-09-20 at 2 50 14 pm](https://cloud.githubusercontent.com/assets/11450276/18684287/9fd1a8c4-7f41-11e6-8f3f-a62b020ee28a.png)

However, when I look at the task for that command in the Dashboard, maxConcurrency display as 500 (see below). I have no idea where that 500 could be coming from.

![screen shot 2016-09-20 at 2 52 08 pm](https://cloud.githubusercontent.com/assets/11450276/18684354/df184fec-7f41-11e6-9f81-bd3ec5deffbd.png)
jeffpeiyt commented 8 years ago

hi @agustintorres , I just realize that probably you did not use the master branch (master branch did not have the dashboard and many features in distributed manner); but use the distributed version branch: as the default concurrency is set at 500: check here The distributed version has more complexity in setting the concurrency when dispatch jobs to slave nodes etc.

Could you please try the master branch, it has less feature but I just tested the concurrency changes is working in display.

The distributed version was not completly tested in production and we suggest you use the master branch version for reliability.

agustintorres commented 8 years ago

@jeffpeiyt: Thanks for the insight. I gave Parallec a try and decided it fits my use case better, so I will be using it for now. Feel free to close this issue.

jeffpeiyt commented 8 years ago

Thank you @agustintorres ! Please feel free to raise issues in parallec if you have more questions.