[MTurk] QA Data Collection Demo Broken

[ethanjperez]

Thanks for your MTurk setup! It's definitely helped me get Turk model evaluations running quickly.

Just wanted to point out an issue I ran into yesterday (3/22) with the nice qa_data_collection MTurk demo. On a fresh ParlAI install, this demo successfully launches the MTurk webpage, but loops infinitely within this while loop, even when I start the task. I think the issue is that it never finds that a valid_agent has started the task. Thus, I just see a gray screen with no text or task.

I have reproduced this issue on 3 separate machines (1 Mac, 2 Linux) and 2 browsers (Chrome and Safari).

Specifically, I run:

conda create -n ParlAI python=3.6
conda activate ParlAI
git clone https://github.com/facebookresearch/ParlAI.git ~/ParlAI
cd ~/ParlAI; python setup.py develop
python parlai/mturk/tasks/qa_data_collection/run.py

And here's an example output:

[ optional arguments: ] 
[  datapath: ~/ParlAI/data ]
[ Mechanical Turk: ] 
[  allowed_conversations: 0 ]
[  block_qualification: None ]
[  count_complete: False ]
[  disconnect_qualification: None ]
[  hard_block: False ]
[  heroku_team: None ]
[  hobby: False ]
[  is_debug: False ]
[  is_sandbox: True ]
[  local: False ]
[  log_level: 20 ]
[  max_connections: 100 ]
[  max_hits_per_worker: 0 ]
[  max_time: 0 ]
[  max_time_qual: None ]
[  min_messages: 0 ]
[  mturk_log_path: ~/ParlAI/logs/mturk ]
[  num_conversations: 100 ]
[  reward: 0.05 ]
[  task: None ]
[  unique_qual_name: None ]
[  unique_worker: False ]
[  verbose: False ]

You are going to allow workers from Amazon Mechanical Turk to be an agent in ParlAI.
During this process, Internet connection is required, and you should turn off your computer's auto-sleep feature.
Enough HITs will be launched over time up to a max of 1.1 times the amount requested until the desired number of conversations start.
Please press Enter to continue... 

Querying the parlai website for possible notices...
Setting up MTurk server...
Heroku: Collecting files...

> fsevents@1.2.4 install ~/ParlAI/parlai/mturk/core/heroku_server_a258e47e-qadatacollection/node_modules/fsevents
> node install

[fsevents] Success: "~/ParlAI/parlai/mturk/core/heroku_server_a258e47e-qadatacollection/node_modules/fsevents/lib/binding/Release/node-v64-darwin-x64/fse.node" already installed
Pass --update-binary to reinstall or --build-from-source to recompile
npm WARN parlai-mturk-task-compiler@1.0.0 No repository field.
npm WARN parlai-mturk-task-compiler@1.0.0 No license field.

+ custom_built_frontend@1.0.0
added 745 packages from 353 contributors and audited 8285 packages in 5.76s
found 0 vulnerabilities

npm WARN bootstrap@4.3.1 requires a peer of popper.js@^1.14.7 but none is installed. You must install peer dependencies yourself.
npm WARN parlai-mturk-task-compiler@1.0.0 No repository field.
npm WARN parlai-mturk-task-compiler@1.0.0 No license field.

added 2 packages from 3 contributors and audited 8287 packages in 3.286s
found 0 vulnerabilities

> parlai-mturk-task-compiler@1.0.0 dev ~/ParlAI/parlai/mturk/core/heroku_server_a258e47e-qadatacollection
> webpack --mode development -q

Browserslist: caniuse-lite is outdated. Please run next command `npm update caniuse-lite browserslist`
Hash: 5d8f0fa7a305489a4831
Version: webpack 4.19.1
Time: 2800ms
Built at: 03/23/2019 11:34:20 AM
                  Asset      Size  Chunks             Chunk Names
server/static/bundle.js  3.19 MiB    main  [emitted]  main
Entrypoint main = server/static/bundle.js
[0] util (ignored) 15 bytes {main} [built]
[./node_modules/webpack/buildin/global.js] (webpack)/buildin/global.js 489 bytes {main} [built]
[1] util (ignored) 15 bytes {main} [built]
[2] ./streams (ignored) 15 bytes {main} [built]
[3] ./extend-node (ignored) 15 bytes {main} [built]
[./dev/app.jsx] 11 KiB {main} [built]
[./dev/css/custom.css] 1.06 KiB {main} [built]
[./dev/css/style.css] 1.06 KiB {main} [built]
[./dev/main.js] 257 bytes {main} [built]
[./node_modules/css-loader/index.js!./dev/css/custom.css] ./node_modules/css-loader!./dev/css/custom.css 352 bytes {main} [built]
[./node_modules/css-loader/index.js!./dev/css/style.css] ./node_modules/css-loader!./dev/css/style.css 402 bytes {main} [built]
[./node_modules/encoding/lib sync recursive] ./node_modules/encoding/lib sync 160 bytes {main} [optional] [built]
[./node_modules/webpack/buildin/module.js] (webpack)/buildin/module.js 497 bytes {main} [built]
    + 462 hidden modules

WARNING in ./node_modules/encoding/lib/iconv-loader.js 9:12-34
Critical dependency: the request of a dependency is an expression
 @ ./node_modules/encoding/lib/encoding.js
 @ ./node_modules/fetch/lib/fetch.js
 @ ./dev/app.jsx
 @ ./dev/main.js
Heroku: Starting server...
 ›   Warning: token will expire 03/12/2020
 ›   Use heroku authorizations:create to generate a long-term token
Creating ethanperez-a258e47e-qadatacoll... done
Enabling http-session-affinity for ethanperez-a258e47e-qadatacoll... done
Scaling dynos... done, now running web at 1:Free
MTurk server setup done.

Local: Setting up WebSocket...
WebSocket set up!
Creating HITs...
Link to HIT: https://workersandbox.mturk.com/mturk/preview?groupId=3ZNZSREO6CCN3O92XRH1S130YZBZXO

Waiting for Turkers to respond... (Please don't close your laptop or put your computer into sleep or standby mode.)

[JackUrb]

Thanks for flagging this @ethanjperez! A bug small bug was indeed introduced in #1541 that I have a fix for in #1579.

[ethanjperez]

@JackUrb great thanks for your help!