Less restrictive shut down policy for ext. 2.0x

[JPierreS]

This is more of a feature request for future versions of the extension.

Would it be possible to make the idle to shut down less restrictive ? Some testers feel they should not be enslaved / glued to the screen all the time. This would allow those who are temporarily AFK but still nearby the computer to receive jobs. Since idle fallback level 1 is removed from 2.0 and it reverts directly to shut down after 3 minutes, perhaps increasing the limit to 6 minutes ? or something more reasonable.

[zfonck123]

yes each 2minutes, is slave job e.e, dont have time for cooking, or take a shower for waiting jobs

[JPierreS]

Well if you are taking a shower, it would not be too convenient to receive jobs, that is clearly an afk and good example of idle policy to good use - same for cooking :-)) But for those who are not active on the computer but still in the same room doing other chores nearby it would be helpful to have a more reasonable shut down limit.

If I am in the shower or cooking, generally takes longer than 3 minutes ;-))

[sweeneyapps]

I have better idea.. let set it to 5 minutes idle time. After 5 minutes, of course the extension will stop polling for work. Whenever you re back on the computer, the extension should automatically become active. It's easier so nobody forgets to check if it's on or off. I think it'll be fair for everyone.

[federicoall]

Yes 5 min is more reasonable.

[JPierreS]

@sweeneyapps I agree, 5 minutes however not so sure about the automatic resume. There are times when a person might be back at their computer but not necessarily want to work. From what I see, we might be able to configure different options of our extension directly from the tester profile page, including sound and other. The auto resume should be optional preferably.

Also in 2.0.x extension starts polling when chrome browser opens - it should always be set to off as default, as with previous version.

What do you think about the code being external now - defeats the purpose of open source. I had made some good mods to my 1.2.10 - not possible anymore with 2 or for contributors to mod and test their code locally.

[sweeneyapps]

@JPierreS yeah, polling is temporary for 2.0. Websocket job distributions system is in the work. I'm hoping that they will find a way to make 2.0 available to any developer. I would suggest RF to develop a mock RF servers so we can test the websocket and polling interaction w/ the extension. It'll be fun debugging it.

[JPierreS]

@sweeneyapps It is my understanding that the new job distribution system will be backward compatible with polling for those who cannot do WS. As far as the pushing of jobs through web sockets some people are having doubts in the community - I myself have few doubts - will this mean generally less jobs and more dilution effect - if not used properly could mean far less jobs and some good testers might be caught in the crossfire. How will the polling interact with WS, those who cannot establish a WS will revert to old system ? Will they be able to get more hits than those who are getting hits through WS ? What about other security issues and concerns / rationing, etc.

As far as 2 the code is external, however one could still mod it to use local code, but one would be stuck in using old code which might not work in some future.

I'm wondering if the externalising of the code and the whole push thing is to counter the abuse and "hack" methods (those who persistently get thousands of hits week by week ?) If so, then this could also affect the GOOD workers who get jobs from official sources too ! I have my doubts in that regards. I am a loyal, long time tester,of 2 year and I have only gotten 100+ hits 3 or 4 times. Lucky if I can get even half that. It is established some people use methods to get 100-200 jobs DAILY, I don't know how that can even be possible, and doing 8 jobs at once and other things I heard about, and obviously with locking of the code and WS / disabling polling this will put an end to this, however those who legitimately get a link from an extension and refresh it will be punished too and those like me who want to do legit mods to their ext also get punished. Again the actions of the bad crowd impact the good crowd. :-))

[shosti]

I'm definitely open to raising the idle limit and also not turning off when the screen is locked (which I think is causing some trouble for people on windows). The situation we're trying to avoid is people going away from their computers and a job getting assigned and abandoned. For now, I think just raising the limit to maybe 8 minutes (give or take) seems reasonable. Auto-resuming also seems reasonable (maybe with a notification double-checking that they want to keep working?).

@sweeneyapps I agree that it's a bit unfortunate that the "backend" of the code isn't open source. I'll discuss internally whether we can open-source a "fake" version for development purposes (it might take a little while though--it's tricky because there are a fair number of moving parts).

@JPierreS w.r.t. backward compatibility, we are definitely moving to a websocket-only approach in the near future and dropping compatibility with the old plugin, which will mean that local mods may stop working. There are 2 main reasons:

1. Scalability--our current backend code is not really able to handle thousands of concurrent testers polling, which is why we currently have to implement aggressive rate-limiting (which nobody likes).
2. Fairness--this actually goes beyond avoiding abuse and is more about assigning work to our top-performing testers more often (which we're not currently able to do). The overall throughput will stay the same, so worries about there "being less work" are unfounded.

If you think you can't use websockets for some reason let me know (there shouldn't be any compatibility issues with reasonably recent versions of Chrome). Overall though, I think the benefits will far outweigh any drawbacks.

The reason we externalized the code is so that we can push releases more often and make sure everyone's using an up-to-date version of the plugin. Again, I think the benefits of more frequent releases and feature updates negate any possible drawbacks.

[JPierreS]

Thank you so much for this detailed reply @shosti - I love those detailed replies - it reminds me of .someone ;-))) I agree about the auto resume with notification if we want to resume work, this is great indeed ! As far as my concerns on the WS front, you raised a good point with argument 2, however it only makes my concerns stronger - Right now the majority of jobs being assigned contain instructions that are not following our tester rules - many of us in the good crowd are reporting those and doing a good job but rarely do we see our bug count go up which suggests a cheating problem or blatant disregard for rules - problem with this is that bug count not going up means frequent algo rejections, and whilst those do not affect our tester accounts, it is still vague as to how you judge a good worker from a bad one, as right now many of us good workers are in the cross fires and my concerns is that with WS, is a scenario where it is not used properly and where good testers are unfairly "categorised" and would get less work as a result of being paired frequently with bad crowd.

Also we are all humans still right ? (Although some days we can have a debate about this ;-))) but for the most part we are humans and imperfect and subject to making some mistakes - and learn from them. So this whole WS creates uncertainty as to how testers are judged - as the current situation shows some unfairness and bias in some areas in regards to rejection - there is still lot of confusion over rules and some difficult scenarios - some testers are not malicious and WANT to get on the right track, how would those be judged against actual malicious testers who pass obvious rules like quotes checking etc.

My concerns about WS remain the same - if this is used as a tool to seed out the bad crowd from the good, given the current situation is this the best move ? will some of us get caught in the cross fire ? I hope I am wrong lol.

BTW, so with this WS will this mean we are guaranteed to get a LIVE HIT (are hits actually reserved) or does it just push a link like the current method ?

I just hope for a future where the jobs will be fair, the instructions fixed, yes worker will be banned for good and including future lives as well and that we can brag about getting so much work :-)) I am still having lucid dreams about being able to complete large jobs without reporting due to matching quotes or other rules policies ;))

[sweeneyapps]

@shosti I do like 2.0. thank you for all of your hard work on this and hopefully you get all the bugs tackled off your long list. Frequent releases is a huge bonus because last time, I tried to submit Pull Request on a very bad bad polling bug. My code got released in about 2 weeks! I can't imagine a lot of tester not knowing that they re losing a lot of opportunities on getting work. Nobody notices it and I had to edit the code locally via the console and debug it until I find the bug. Anyway, I'm sure everyone will be happy w/ 8 minutes and on the chrome version, so far I know of is that Google stopped supporting Chrome on some OS:

"Windows XP, Windows Vista, OS X 10.6, OS X 10.7, and OS X 10.8 are no longer supported." (arstechnica.com, April 2016)

@JPierreS I did try to stay up 24hrs from 5pm to 5pm to prove a theory that 100 jobs are impossible! I made it to 98 jobs! Of course I only get 10% instead of 20%. I did try bring it up on AMA recently:

Any plan on changing the bonus rate? Such finishing 100+ jobs for 20% bonus. All jobs are not equal so why not set it to 500+ steps completed for 20% bonus?

• This is also a question we answer every month. There’s currently no plan to change the daily bonus rate or increase payments for each step. If there ever is a plan for this, I’ll let you all know!

of course that's not the answer I'm expecting... I was expecting for my feedback to be recorded and discussed w/ RF team or higher up. It'll be beneficial for all the testers working so hard and greatly be rewarded. :+1:

I'm sure RF got a smart team of engineers planning an attack on cheaters :) It'll be fun if RF put up fake work assignment and everyone thought it's normal job.... but nope! Those happily trigger YES tester would be busted ASAP! and ofc double checking, triple checking w/ fake job site that looks so real job.. 3 strikes you re OUT! PNG!

I think that some testers are using translator for the jobs and I have seen some testers asking question about rejections and not understanding those rejections when it's clearly a good rejection. For example, some of them focusing on quotation to be exactly as shown on website. But we do have to read the question in full sentence to apply what's the author is looking for in the quotation.

All I'm hoping for RF to recruit some good quality testers and increase payment per steps to make it sense for everyone. Also yup hopefully the system improve and go smoothly but we can't always predict what the system will do.

[sweeneyapps]

@shosti @JPierreS here's what I did for fun 2 months ago.

https://github.com/sweeneyapps/JobCheckerForRainForest

I created a fake server and testing my theory on how fast the extension should open a new tab after accepting a job. I did this because of older extension didn't open new tab in time when it turns YES. With this test, I discovered that extension doesn't delay on opening a new tab if the server is still loading a page.

[JPierreS]

@sweeneyapps - In regards to your first post, do you mean the polling bug that caused the extension to stop polling due to an unexpected json ? Yeah assuming that most testers are not too technical yes, this explains why some testers complained of not getting any hits. A quick fix for that was just resetting the extension. I think these new versions should go through beta testing before being released to the wild. V2 for some testers is not working as intended - some are reporting that the green icon is turning gray after few seconds, forcing them to constantly click to turn back on.

Yes from 3 to 8 minutes that is a great jump - How about a dynamic policy - if tester has too many consecutive abandons reduce policy (with a floor value of 3) and a ceiling value of 10, as a tester accepts more jobs in a timely manner.... guess that is asking too much lol.

In regards to your suggestion about bonus based on number of steps and not number of hits I completely disagree with that - This would only encourage bad work. Do you believe that would help the YES workers situation if RF gave a bonus based on total steps and not actual # of jobs ? I think this would backfire badly. Let's hope that with WS in place the PROPERLY proven bad workers get less work and the good ones get more, so maybe 100+ hits a day will be possible after all, and they will have to up the bonus to 200 o 300 hits for the 20% instead ! (evil laugh) lol. As far as the fake work idea I've been suggesting this for years lol I'm not talking about the obvious "training jobs" that everyone knows how to rig + there is an exploit some tech users can use to get answers to jobs configured with expected answer results. So the fake jobs idea is brilliant, those would guarantee to be checked and catch bad workers. A fake job for every possible rule (quotes, pop-ups, position, etc.) love it, I think this would catch more bad workers much faster, and continually changing them so the testers don't learn that they are trainers!

I agree about the translators, just read some of the GITHUB topics that are clearly google translated some are hilarious ! One of them refers to whether they will keep getting paid from the tropical forest and something about google being hairy lol.

I fully agree with you about recruiting quality testers / and paying more we just have to keep working hard to convince them to lol.

As far as your find on the delayed link, great find ! Very helpful. I guess this is one of the perks of open source.

[sweeneyapps]

@JPierreS ahh let me find the Pull Request that is already merged and you can look at the changes I made. Then the 2nd pull request on Unexpected JSON didn't go thru so I closed it when 2.0 came out.

here's the first pull request https://github.com/rainforestapp/tester-chrome-extension/pull/364

you got a good point on steps idea. The way I see it is that how do we feel if we sat in the chair for 8 hours and actually did only 5 of 100 steps job? I'll scream! lol But I'll not complain anyway because we re still getting 500 steps times .04 = $20 :) Btw, WS is a good start but I can understand your doubt into that system on how Rainforest will evaluate all testers, who get the most job or not. Test Interface is still slow and I don't think we'll ever reach 100 steps ever again!.. I never did. (almost)

I agree on beta testing, It'll be nice for RF to recruit maybe 5-10 beta testers. Those testers will have access to staging part of the development using VPN. I'll love to join! But the current setup on staging server probably doesn't produce errors to test on. There's a lot of possible factors on why bugs occurred.. For example, some testers re using windows XP. <--- I think it's time to upgrade :)

[sweeneyapps]

@JPierreS the bug I discovered on first pull request was error on finding workTab (tab). When the server say Yes to job, the extension try to refresh the workTab (old) but that workTab is closed! So extension just updated the workTab to null then drop the job! It goes on Odd and Even game on getting new work.

[qpPykT]

@JPierreS @sweeneyapps Easy, guys. About the testers who are using google translator I think we have more than 70 percent of all rf testers who using it. I use it for writing notes or just writing something because my writing level in english not so good than I would like to have. Recruiting "testers" from buxes is not right place for that. Here almost all of "testers" have low level a little bit have middle level and maybe a few of us have height level of some kind monkey testing. We just do test cases and nothing more.

[huynhxuanvinh]

I just want the next update will erase the status "working" of the add-on when it opens a new tab but the testers have to close it ( because another worker takes it or something else). I see the old version works very well and don't know why people upgrade it to an awful version like this ( really sorry to author of this extension but I have to say that). I meet 10-20 "another worker...." message a day and have to disable add-on and restart it manually or close chrome and open it again because the add-on with text "working" will never open a new hit.

[sweeneyapps]

@qpPykT JPierreS are just discussing something else about testing bug. It has nothing to do with current tester's level of knowledge of programming or something else.

We are talking about a group of testers who will be willing to test for bugs on new release of extension before it is released into real group of testers. It is like a step of process to test extension to make sure it is ready for all testers to use. That way, less bug for all of you.

I am sorry if we offend you on google translator but we are just finding it funny how google translator work because it is not always 100%, We know that. again, I'm sorry.

[sweeneyapps]

@huynhivan don't worry, all the bugs will be fixed soon. Sometime, it's part of the process when new program came out, and it does have bug. RF is working hard to fix all the bugs soon as possible. We do need this new version because there will be better system to assign work to all testers. The new system is better and you won't get "another worker" error anymore.

[huynhxuanvinh]

@sweeneyapps I know, everyone has worked hard to make the system better day by day and I also appreciate the effort of all team. I write these complaints with the hope someone here in the team can notice it and know it's the problem of many testers so they will update it in the next version. Thank you for your comment.

[JPierreS]

@sweeneyapps

• in regards to a beta testing team; There is already one in place ;) Also they have the super testers team with super natural abilities they can do the impossible lol !
• in regards to the 500 steps idae; In some days that would be easy to reach, especially with all those large jobs circulating, like certain NDA clients and their 88 step jobs or another customer that releases those 100-157 jobs lol, but even worse is that not a single time that I report those impossible to finish 100+ steppers do I get bug count, so the yes workers are enjoying those hits, do yes workers deserve even more money than they are making now ? No. I think GOOD work should be rewarded GOOD/SKILLED testers should be rewarded, not the actual number of jobs/steps completed - it is the GOOD/SKILLED workers that are beneficial to RainforestQA's success - not the "Job Racers" who want to race to the finish line and brag to their mates how much they earned big today because they passed all the 156 steppers they received.. On that note, I'd love a $20 bonus for 500 steps completed, I'd probably never end up getting one as I keep failing most tests nowadays due to quote matching rules, failures or other - If I got a bonus on each failed test I'd be earning 3 figures daily bonuses lol !
• google translator; Of course the fun part was not made at the user itself but the results returned from the translator itself. As far as tester's native languages, well this can be an issue - I think to effectively complete this task and provide quality results, a tester should have minimal comprehension of english reading and writing to fully understand what is being asked - Using google translator might not be optimal as proved already. If testers who are not fluent enough in english don't understand the instructions or do not work properly this is not good for anyone - this would NEVER pass with competitor sites that do testing. The quality of work should never be affected by this. We all care for this task and those who do want to preserve it, if they are expected to grow and give us more work, they can only do this if they can provide quality results to their clients, and that their existing ones are happy and renew and hang around - they won't if they get sloppy results for whatever reason.

As far as PWS, so no more "another worker" - So work gets assigned and reserved first then pushed right ? So would this effectively eliminate no_vm errors ? In other words a link is passed only if it is a live job, correct ? Perhaps this is where people who were getting mostly oops links from ext and refresh them might notice less jobs, as the current situation is that the ext does not always deliver live jobs but mostly oops, no vm or something bad links ;)

[mariaelite]

that was friendly form them put the discriminative element for those who use google Traslate right ?, but in any part of his commentary mentioned that they do only by selfishness. I am particularly interested in the opinion Rainforest team, not the other user who just wants to hit stop more, just not sincere and does not say it directly.

[richardfuca]

If testers using google translate and then doing the test wrong because of a bad translate is an issue, a possible fix would to be detect Google translate in the training jobs (not regular jobs, training should be the gold standard, if a normal test needs translate it's a bad test), and give rejections if it is detected

[Electroglitch]

The most ironic thing i saw once was forum post that goes something like: "What is bugs?" XD

Oh yeah and other similar : "What is maximum number of bugs?"

[JPierreS]

Some people here need to understand that it is not a question of discrimination ! Remember that Rainforest is a business they have a mandate to deliver quality results to their clients. This task should be reserved to testers with enough english reading/writing skill to be able to effectively complete a job and submit good results. As far as this section of the site it is not just opened to RF staff to respond but other users - many are long testers and skilled and provide accurate responses.

I think the "discrimination" card is overly used - nothing justifies bad work. If a tester is submitting bad work because they do not understand the instructions or the translator is returning results that are inaccurate then they should not work this task - It seems to me that some testers don't understand the implication of just "PASSING" a step that needs to be failed or taking this work seriously. So there is no discrimination or selfishness, some of us are hard working, skilled tester part of a chosen trusted team of testers and have a right to comment on what is observed as well.

The use of google translator might cut i with some CrowdFlower jobs, but not for this task IMO. Basically translating already bad instructions using a google translator is recipe for disaster, might cut it for certain jobs but not all. I agree with helloworld 100%

Locking this topic.

I think moving forward RF should seriously considering making this a closed tester thing and very carefully selecting testers, harbouring only the skilled tester, bad workers or malicious workers should look for another task, regardless of the reason - it's not a question of discrimination but skill / ability. It is clear by reading the forums and the questions asked that some don't understand the task and the rules and just blindly work without as much thought on the implications. If you apply for a job for a company and get hired and do sloppy work and you get fired, will you use the "discrimination" card ?

I've done my own research and experiment and used google translate on some hits to translate to other languages I know and the results were absolutely HORRIBLE, I wonder how the heck some people could possibly complete hits this way. Just ignoring the obvious and clicking YES doesn't cut it. Maybe Rainforest is too lenient about this, but some of us rely on this supplement to income for whatever reason and don't want to see the task destroyed because of bad workers, just like a great number of CF went down for the same reason.