Ability to autobench + upload results?

[Kramerican]

It would be epic if a simple comparison portal could be made, something like (the now defunct?) serverscope.io as well as enabling hosting providers to provide systems which would be periodically auto-benched and collated on this portal.

I / we might be interested in contributing to developing something like this, as existing comparison sites out there are not very vps focused and/or are broken/obsolete in various ways.

[masonr]

Hey there, I appreciate the interest and the great suggestion! Another YABS user had a similar concept in mind, but I don't believe it has been pursued yet.

I am not opposed to this idea and would gladly incorporate into the project, should something be developed to house the scores for comparison. I will, however, like the uploading of the results to the comparison website to be opt-in rather than opt-out. Meaning a switch/flag (i.e. -u) would need to be present when running the script in order for the results to be uploaded. Wouldn't be hard on my end to modify the script to save off the values that are going to be uploaded to the comparison site (geekbench scores, disk iops/speed results, net speeds, etc.).

Auto-benching could be done very easily without any modifications, simply by using cron. Add something like:

0 */12 * * * cd /home/user/yabs && DATE=`date "+%F-%T"` curl -sL yabs.sh | bash >> yabs_"$DATE".log

to the crontab to bench every 12 hours and save the results to a log file. Obviously, this can be changed if the comparison site is completed and the upload flag can be put in place instead of saving the log file.

If you decide to pursue the comparison website idea, feel free and keep me updated on the status! Unfortunately, I can't really commit to helping out with the project due to time constraints, but I'll always be available to bounce ideas off of or to provide feedback, if desired.

[webdock-io]

That's great news! I was using my "personal" account for the previous post, which is why I didn't see you had replied here till now. But as you can see we are a Cloud hosting provider.

Our core interest is of course - without a doubt - to be able to show users in an objective and "as unpartial as possible" manner what our system scores are as well as enabling users to accurately - and fairly - compare providers across metrics and cost. It takes a lot of time to do manual benchmarking and setting up reports like this, and we have sort-of given up on doing that for our website as it is simply too costly in terms of man-hours.

So, that's our motivation. The other motivation is that the existing comparison sites out there focused on Cloud Hosting are, well, quite bad. Some (will not name names) are for-profit and this gets in the way of usability (sponsored providers are "in the way" of accurate comparisons) while others are simply not maintained / defunct.

We propose the following:

1. We would come up with a site design and comparison views / logic which we feels serve the customer best (quick, accurate overview of providers based on metrics they are interested in, as well as cost, if known)

2. As the script / data provider we would defer to your judgment as the fair arbiter of what information should be shown and how

3. We would, once launched, invite all players in the industry to "sign up" and set up automatic benchmarking of their instances, which will be made available for free, for everybody.

4. We would commit to 5 years of maintenance of the site. If the site becomes some sort of industry standard (we can only hope) - then we would convert this to a legal not-for-profit entity, hopefully ensuring unbiased reporting of price/performance throughout the hosting industry for years to come

We agree reporting should be opt in. We also see that your end of this would be very simple - just submit all the data as json, adding in a few fields so users can define a provider name, plan id/name and plan cost.

We imagine we would be able to combat fraudulent submissions with the power of crowdsourcing - i.e. we would suppress test results which are statistically anomalous compared to what has been submitted. Details of this is something we will try and hash out in a sane manner :)

Can you send me a PM with contact details, so we can discuss further? With a positive reply, I will get the design and technical process underway on my end.

[masonr]

@webdock-io - I owe you a response to your email that you sent awhile back so we can talk specifics about this in case you are still interested in pursuing this idea. I dropped the ball on formulating a plan and replying to you quickly, so my apologies on that front. Be sure that this is still on my mind and I've been thinking about it for some time now. Talk soon!

[plerohellec]

I gave it a shot at vpsbenchmarks.com. It's very fresh, just deployed. New Yab. I tried to keep it simple, you copy and paste the yabs.sh output in the box, add a provider name and a plan name and you get a page with the data in nice html tables plus a couple of charts. There's also a list of recent yabs runs.

The next steps will be to allow sorting and filtering in the list and probably a comparison page. The feature is wide open right now so I expect it will be abused, at that point I'll probably have to require registration to submit new data.

Please give it a try if you get a chance and let me know what you think.

[masonr]

@plerohellec Awesome, nice work! Looks really nice and easy to use. :)

I've decided that I'm going to bake in a way to send the YABS results to any URL passed as an argument. Details are still a little up in the air, but the end goal is to have a way to send the results in json form to any benchmark collection site that can intake the data. With multiple people working on separate sites to pull in, organize, and display YABS scores, I'd rather not be the one to designate a single place to send the results to. I think this will be the best way to democratize things so that any third-party site can be used by the script.

Perhaps this is a feature you could add to vpsbenchmarks.com. I'm open to any thoughts or comments on this approach.

[plerohellec]

That's sounds great. I'll be glad to adapt or create a new endpoint at vpsbenchmarks.com for the format you come up with. Currently, you can post the yabs.sh output to VPSB with:

curl -sL yabs.sh | bash > yabs.out
curl -F "data=<yabs.out" -F "os_version=`lsb_release -s -d`" https://www.vpsbenchmarks.com/yabs/upload

So it's just a POST with a data and an optional os_version field.

I'm particularly thrilled about getting the data in json format!

I have one additional wish: have the ability to pass an api token as an option to yabs.sh along with the url so that the server can limit abuse if need be.

[webdock-io]

I applaud this initiative and I apologize for our inactivity on this matter - we fully intended to contribute here at some point, but as our platform grows a lot of other stuff has taken precedence.

While I applaud this development I think a prudent step would be to reach out to the various VPS benchmarking portals and make them aware of this new feature in YABS in case they want to adopt it.

I thank @plerohellec for taking the lead here, although I am still not happy about the fact that vpsbenchmarks.com refused to list us as they thought we were being unfair to the competition :) Maybe they will reconsider once we have our dedicated VPS product online (sometime this year we hope) which should then be more of an apples-to-apples comparison with the traditional providers.

Let us know if we can contribute in any meaningful way here.!

[plerohellec]

@webdock-io ah yes, I remember your last email. It's been a while, I can take another look if you'd like, please contact me directly, you have my email address.

[masonr]

@plerohellec -

I have one additional wish: have the ability to pass an api token as an option to yabs.sh along with the url so that the server can limit abuse if need be.

I'm pretty sure this can be achieved outside of the script (via the URL directly), but I'll need to test. Imagining something like this:

curl -X POST https://yabs.sh/upload/json?apikey=abcde12345fghij67890
   -H 'Content-Type: application/json'
   -d '{json}'

so, the end user would call something like:

curl -sL yabs.sh | bash -s -- -u "https://yabs.sh/upload/json?apikey=abcde12345fghij67890"

---

@webdock-io -

I applaud this initiative and I apologize for our inactivity on this matter - we fully intended to contribute here at some point, but as our platform grows a lot of other stuff has taken precedence.

Hey, no worries! I completely understand. Life has a way of throwing constant curve balls your way, especially when running your own business. I think this issue being opened and some of the discussion around has been the catalyst for some of the yabs benchmarks sites that are currently being worked, so thank you nonetheless.

While I applaud this development I think a prudent step would be to reach out to the various VPS benchmarking portals and make them aware of this new feature in YABS in case they want to adopt it.

Good idea! Will try and get the ball rolling on that soon.

@s0c-org has been developing a YABS result aggregator for some time now and we've had a lot of back and forth on the subject. Beta version is here if anyone reading this is curious and would like to check it out: https://debug.s0c.org/. Working him to deploy a test branch of YABS to shoot the json results to his site so we can grease the wheels and get things moving along. Eventually, once the functionality is complete, it'll get merged into the main branch and allow any site to be used (main ones will be listed in the Readme so that users are aware). More to come soon.

[plerohellec]

@masonr I think you're right about the api key as a query parameter, that should work.

I'd be happy to help you test yabs.sh with json output if it's available somewhere too.

[masonr]

@plerohellec -- sorry for the delay addressing this! I just committed some changes to the repo to JSONify the YABS results and upload them to a benchmark website of choice.

curl -sL yabs.sh | bash -s -- -j "http://example.com/yabs/post"

So you can know what to expect, an example JSON output can be found here: https://github.com/masonr/yet-another-bench-script/blob/master/bin/example.json

Give me a shout with any feedback!

[webdock-io]

@masonr Although we don't have an immediate use case in mind for us over here at Webdock at this time, this is a pretty epic feature and the json looks yummie! Great work! Thanks for moving on this

[plerohellec]

Thanks @masonr, that looks awesome! I think this json output will be hugely useful.

One wish though: I think the output format feature should be decoupled from the upload feature. The json output format is by far the most useful part of those changes and it has value even without the upload. The upload is not a "one size fits all" and it can easily be implemented outside of yabs.sh just by piping yabs.sh output. For example, some servers will expect additional parameters, the VPS provider name, authentication parameters/headers or the desired response format (Accept header) but if it's hard coded in yabs.sh, the upload can't be customized. I'd recommend 2 separate options: "-j" or "-f " for specifying the output format and "-u for performing the upload for example.

Not as important but I wonder if units and labels should be added to the json output especially for fio, I'm afraid there is a risk for confusion with just ["4k", 154305, 38576, 154712, 38678, 309017, 77254].

Does that make sense?

[masonr]

Thanks for the feedback, @plerohellec!

One wish though: I think the output format feature should be decoupled from the upload feature.

Totally doable and I understand the benefits of doing it that way. I'll work that in shortly when I push some updates.

Not as important but I wonder if units and labels should be added to the json output especially for fio

Yes! Didn't feel like writing up the schema or annotating last night when I cobbled all this together, but yes, desperately needs either more explicit naming in the JSON format, or a companion data schema doc that explains how to interpret all of the fields.

I'll work these changes in over the next couple days and get an update out for some further feedback. Really appreciate your insight and help! :)

[masonr]

@plerohellec just to follow up on this.

I just pushed another update to the JSON-related capabilities. Following your advice, I added a couple additional flags:

-j will print the JSON results out after the script is done running, thus can easily be piped (i.e. curl -sL yabs.sh | bash -s -- -j | tail -1 | <your_cmd_here> -)

-w will write the JSON results out to a file in the current directory (named yabs_{date}-{time}.json)

-s will upload the JSON results to the passed URL. This now also accepts multiple URLs to send the results to multiple websites simultaneously

I touched up the JSON schema as well to make things more explicit. I will be making a companion schema definition soon to define what units are being used for the various fields and all that good stuff. Let me know what your thoughts are!

[plerohellec]

Thank you @masonr , this is a good, flexible set of options.

For the -w, I'd rather have the yabs.sh user choose the filename himself with -w <filename> because if yabs.sh builds the filename then the user will have to guess this filename/date format when he wants to use it and that could be a frequent source of errors.

For fio units, I wonder if the simplest way would be to have them in the json output in the same fields as the values, the same way you did it for the iperf3 ("122 MB/s")?

[masonr]

@plerohellec -

For the -w, I'd rather have the yabs.sh user choose the filename himself with -w because if yabs.sh builds the filename then the user will have to guess this filename/date format when he wants to use it and that could be a frequent source of errors.

Changed up the -w flag to accept a file name (or path + file name)

For fio units, I wonder if the simplest way would be to have them in the json output in the same fields as the values, the same way you did it for the iperf3 ("122 MB/s")?

I personally think that it'd be best for all the units would be in the raw form provided by the tool in whatever base units the tool provides. Fio outputs the speed results as KB/s by default. Unfortunately for iperf3, I'm have to grep the output which doesn't provide a raw measure in a base unit (only formatted [i.e. '122 Mbits/s']), so I'm stuck with the formatted value.

[plerohellec]

Thanks for the -w change.

How about add a speed_unit field at the top of fio? Something like this:

"fio": {
  "speed_unit": "KB/s",
  "results": [{
        "bs": "4k",
        "speed_r": 165590,
        "iops_r": 41397,
        "speed_w": 166027,
        "iops_w": 41506,
        "speed_rw": 331617,
        "iops_rw": 82903
    },
  ]
}

[3xploiton3]

FYI For alternative

https://yabsdb.com/

curl -sL yabs.sh | bash -s -- -s "https://yabsdb.com/add"

[plerohellec]

@masonr, I finally implemented the json parsing and uploading with yabs.sh -s. So you can now run curl -sL yabs.sh | bash -s -- -s https://www.vpsbenchmarks.com/yabs/upload and visualize the results at https://www.vpsbenchmarks.com/yabs.

Some more feedback on the json format now that I'm very familiar with it:

• There are some fields in the text output format of yabs.sh that are missing from the json. Those that I miss the most are Host, ISP, Location, Country and VM TYpe. They are all very useful for adding context to the test and organizing the yabs data. I would love to have them in the json.
• I still miss units in the json, not only fio but also mem/ram, swap and disk. The problem with the lack of units is that, if either yabs.sh or fio decide to change them for any reason, services that parse the yabs.sh json will have no way to know of the change and will record the data with the wrong unit. I can't think of any solution other than including the unit in the json.
  • I was looking at how fio does it when it prints json (fio --output-format=json) and I see that it includes the unit as a suffix of the metric name, for example io_bytes, io_kbytes or latency_ms. Maybe that could be a solution for yabs.sh too? I'm really open to anything to get those units with the data.
• I was not sure what the os/virt field was? Is it the same as "VM-x/AMD-V" in the text format?

Thanks again for the great work you're doing with yabs.sh!

[masonr]

@plerohellec - awesome! Sounds good and nice work on the json implementation.

Addressing your feedback:

• Missing JSON elements: I'm tracking and will add all the missing pieces to the JSON output very soon. Will push an update that will bring the JSON output 1-to-1 with the textual output (including script start/stop/runtime, vm type, all network info).
• Units: Agreed this is a must to standardize and add to the output. I'd like to not cause any backwards compatibility issues if I push a schema change, so any changes to the JSON structure will just be additions to the existing schema for better or worse. mem/disk and fio will be the low hanging fruit to knock out in the near term. With the eventual goal of getting iperf3 and ping latencies standardized as well.
• Virt: Yes, cpu/virt field is same as VM-x/VMD-V in text form to indicate if the machine is virtualization ready.

As a side note, I'll be adding your site (and others) to the readme as places where YABS results can be uploaded and viewed.

[masonr]

@plerohellec - finally got around to this. Script now outputs JSON in the format here: example.json. Should address the units issue and add in all the missing stuff that wasn't being put in the JSON results. Also, added the sites that accept JSON results from YABS that I'm aware of to the README.

[plerohellec]

Thank you @masonr , this will work!

[masonr]

I'm going to finally put this issue to rest. I think YABS now has a solid way of uploading results and a few really great sites that support JSON result uploads. Appreciate all the input, help, and support to bring this together!

If any other issues or requests come up in the future, feel free to open a new issue for it! Thanks all!