[MTurk] Add support for React Hooks

[pringshia]

Bug description Currently React Hooks don't work in custom frontends for tasks. Attempting to use them results in an "Invalid Hook Call" exception, due to point #3 from this docs page:

3. You might have more than one copy of React in the same app. https://reactjs.org/warnings/invalid-hook-call-warning.html

This occurs because we are bundling two separate versions of react in the browser. The task and the webapp both have their own redundant dependencies.

Reproduction steps Enter steps to reproduce the behavior:

1. For a custom mTurk task, try to use React Hooks, e.g. useState(...)
2. Launch task in sandbox, notice that the task fails by inspecting the web console

Expected behavior Hooks should work

Screenshots

[pringshia]

After a bunch of mucking around, was able to get this to work!

Below are the directions I followed. The idea is to ensure that the task and the core library both use the SAME version of React. Currently, the task bundles in its own redundant copy of React and ReactDOM which aside from this issue, also isn't very efficient.

I think a possible deliverable out of this exercise could be to update the custom frontend example to support Hooks, or adding some sort of documentation, or both. Will follow up with a PR next week.

1. Ensure that the React versions are the same between the package.json in your task file and the package.json files @ parlai/mturk/core/react_server/package.json and parlai/mturk/webapp/package.json (TBH - I haven't figured out the difference between these two yet, perhaps @JackUrb can advise).

2. For your task, ONLY include react and react-dom as a peerDependency. Do NOT have it as a regular dependency or devDependency.

3. Update the task's webpack.config.js to:

{
    output: {
        path: __dirname + "/dist/",
        filename: "custom.jsx",
        library: "built_custom_components",
        libraryTarget: "umd",
        umdNamedDefine: true
    },
    externals: {
        react: "react",
        "react-dom": "react-dom"
    }
}

[JackUrb]

Re 1) this is an artifact of me just trying to get the webapp working, and it existed before the webapp was able to render tasks. Cleaning this up would be awesome but I really have no direction on how to do it.

[josharnoldjosh]

Hello, I am trying to get the react-sketch package to work: https://github.com/tbolis/react-sketch/blob/master/package.json

My steps: Clone the parl.ai repository: git clone https://github.com/facebookresearch/ParlAI.git cd ParlAI python3 setup.py develop cd ParlAI/parlai/mturk/tasks/react_task_demo

I then copy and paste react_custom_with_deps in the same directory and name it gan_draw_2

cd ParlAI/parlai/mturk/tasks/react_task_demo/gan_draw_2/frontend npm install

I check that I can do cd ..; python3 run.py, and it works. I then try to install my custom package.

cd ParlAI/parlai/mturk/tasks/react_task_demo/gan_draw_2/frontend

npm install react-sketch --save, this installs the react-sketch package which can be found here: https://github.com/tbolis/react-sketch/blob/master/package.json

I look at the github for react-sketch in the package.json and analyze what react version is being used. I notice it is 16.6.1, as shown from an excerpt below:

"dependencies": {
    "prop-types": "^15.6.2",
    "react": "^16.6.1",
    "react-dom": "^16.6.1"
  }

I then follow step one as outlined above by @pringshia and update the versions of react and react-dom to 16.6.1

In ParlAI/parlai/mturk/tasks/react_task_demo/gan_draw_2/frontend/package.json

"dependencies": {
    "bootstrap": "^4.1.3",
    "fetch": "^1.1.0",
    "jquery": "^3.0.0",
    "popper.js": "^1.14.4",
    "react": "^16.6.1",
    "react-bootstrap": "^0.32.4",
    "react-dom": "^16.6.1",
    "react-table": "^6.8.6"
  }

In ParlAI/parlai/mturk/core/react_server/package.json:

"dependencies": {
    "bootstrap": "^4.3.1",
    "fetch": "^1.1.0",
    "jquery": "^3.0.0",
    "popper.js": "^1.14.4",
    "rc-slider": "^8.6.3",
    "react": "^16.6.1",
    "react-bootstrap": "^0.32.4",
    "react-dom": "^16.6.1",
    "react-table": "^6.8.6"
  }

In ParlAI/parlai/mturk/webapp/package.json

"dependencies": {
    "bootstrap": "^4.3.1",
    "fetch": "^1.1.0",
    "jquery": "^3.0.0",
    "popper.js": "^1.14.4",
    "rc-slider": "^8.6.3",
    "react": "^16.6.1",
    "react-bootstrap": "^0.32.4",
    "react-dom": "^16.6.1",
    "react-table": "^6.8.6"
  }

I follow @pringshia the second step, I move react and react-dom to peerDependency.

In ParlAI/parlai/mturk/tasks/react_task_demo/gan_draw_2/frontend/package.json, my entire package.json is now:

{
  "name": "custom_built_frontend",
  "version": "1.0.0",
  "description": "",
  "scripts": {
    "dev": "webpack --mode development"
  },
  "main": "./dist/custom.jsx",
  "keywords": [],
  "author": "",
  "dependencies": {
    "bootstrap": "^4.1.3",
    "fetch": "^1.1.0",
    "jquery": "^3.0.0",
    "popper.js": "^1.14.4",    
    "react-bootstrap": "^0.32.4",
    "react-table": "^6.8.6"
  },
  "peerDependencies": {
    "react": "^16.6.1",
    "react-dom": "^16.6.1"
  },
  "devDependencies": {
    "@babel/cli": "^7.1.0",
    "@babel/core": "^7.1.0",
    "@babel/plugin-proposal-class-properties": "^7.1.0",
    "@babel/preset-env": "^7.1.0",
    "@babel/preset-react": "^7.0.0",
    "babel-loader": "^8.0.2",
    "css-loader": "^1.0.0",
    "style-loader": "^0.23.0",
    "webpack": "^4.19.1",
    "webpack-cli": "^3.1.1"
  }
}

Lastly, I follow step 3 outlined by @pringshia, and update my webpack.config.js:

In ParlAI/parlai/mturk/tasks/react_task_demo/gan_draw_2/frontend/webpack.config.js:

My entire webpack.config.js is now:

/*
 * Copyright (c) Facebook, Inc. and its affiliates.
 * This source code is licensed under the MIT license found in the
 * LICENSE file in the root directory of this source tree.
 */

var path = require('path');
var webpack = require('webpack');

module.exports = {
    entry: './components/custom.jsx',
    output: {
        path: __dirname + "/dist/",
        filename: "custom.jsx",
        library: "built_custom_components",
        libraryTarget: "umd",
        umdNamedDefine: true
    },
    externals: {
        react: "react",
        "react-dom": "react-dom"
    },
    resolve: {
        extensions: ['.js', '.jsx'],
    },
    module: {
        rules: [
        {
            test: /\.(js|jsx)$/,
            loader: 'babel-loader',
            exclude: /node_modules/,
        },
        {
            test: /\.css$/,
            loader: 'style-loader!css-loader',
        },
        {
            test: /\.png$/,
            loader: 'url-loader?limit=100000',
        },
        {
            test: /\.jpg$/,
            loader: 'file-loader',
        },
        ],
    },
};

Next, I navigate to my custom.jsx in ParlAI/parlai/mturk/tasks/react_task_demo/gan_draw_2/frontend/components/custom.jsx

I replace it entirely with the following:

custom.jsx:

import React from 'react';
import { SketchField, Tools } from 'react-sketch';

class Test extends React.Component {
    render() {
        return (            
            <SketchField                                    
                style={{backgroundColor:'#f9f9f9'}} width='400px' 
                height='400px'
                tool={"Pencil"}
                lineColor={"black"}
                fillColor={"black"}
                lineWidth={30}                    
            />            
        );
    }
}

var LeftPaneHolder = {
  // default: leave blank to use original default when no ids match
  Evaluator: Test,
  'Onboarding Evaluator': Test,
  Answerer: Test,
  'Onboarding Answerer': Test,
};

export default {  
  XLeftPane: LeftPaneHolder
};

I then run my task by doing python3 run.py

I then get this error from my terminal:

ERROR in ./node_modules/react-sketch/dist/index.js
Module not found: Error: Can't resolve 'jsdom' in '/Users/josharnold/Desktop/desktop/paper2/ParlAI/parlai/mturk/tmp/heroku_server_cebd45ed-gandraw2/dev/components/built_custom_components/node_modules/react-sketch/dist'
 @ ./node_modules/react-sketch/dist/index.js 1:85-101
 @ ./components/custom.jsx

I follow the advice from @JackUrb in #2004 and add:

  node: {
    fs: 'empty',
    child_process: 'empty',
    net: 'empty',
    tls: 'empty',
  },

To ParlAI/parlai/mturk/tasks/react_task_demo/gan_draw_2/frontend/webpack.config.js

I rerun the task python3 run.py, however, for some reason, nothing has worked and I am still left with the same errors. Would it please be possible to have some guidance on how to get this library, react-sketch, working in my project? It works perfectly with my normal node.js server locally.

Here is my final error output:

Joshs-MacBook-Pro:gan_draw_2 josharnold$ python3 run.py 
[ optional arguments: ] 
[  datapath: /Users/josharnold/Desktop/desktop/paper2/ParlAI/data ]
[ Mechanical Turk: ] 
[  allowed_conversations: 0 ]
[  block_qualification: None ]
[  count_complete: False ]
[  disconnect_qualification: None ]
[  hard_block: False ]
[  heroku_team: None ]
[  hobby: False ]
[  is_debug: False ]
[  is_sandbox: True ]
[  local: False ]
[  log_level: 20 ]
[  max_connections: 30 ]
[  max_hits_per_worker: 0 ]
[  max_time: 0 ]
[  max_time_qual: None ]
[  min_messages: 0 ]
[  mturk_log_path: /Users/josharnold/Desktop/desktop/paper2/ParlAI/logs/mturk ]
[  num_conversations: 1 ]
[  reward: 0.05 ]
[  task: None ]
[  tmp_dir: None ]
[  unique_qual_name: None ]
[  unique_worker: False ]
[  verbose: False ]
[ Current ParlAI commit: c062a4c52efdc4cd9d07c67bb48b4d9333b7d76f ]

You are going to allow workers from Amazon Mechanical Turk to be an agent in ParlAI.
During this process, Internet connection is required, and you should turn off your computer's auto-sleep feature.
Enough HITs will be launched over time up to a max of 1.5 times the amount requested until the desired number of conversations start.
Please press Enter to continue... 

Querying the parlai website for possible notices...
Setting up MTurk server...
Heroku: Collecting files... for  /Users/josharnold/Desktop/desktop/paper2/ParlAI/parlai/mturk/tmp
Build: Detected custom package.json, prepping build
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN custom_built_frontend@1.0.0 requires a peer of react@^16.6.1 but none is installed. You must install peer dependencies yourself.
npm WARN custom_built_frontend@1.0.0 requires a peer of react-dom@^16.6.1 but none is installed. You must install peer dependencies yourself.
npm WARN parlai-mturk-task-compiler@1.0.0 No repository field.
npm WARN parlai-mturk-task-compiler@1.0.0 No license field.

+ custom_built_frontend@1.0.0
added 1 package and audited 115 packages in 1.935s
found 0 vulnerabilities

> custom_built_frontend@1.0.0 dev /Users/josharnold/Desktop/desktop/paper2/ParlAI/parlai/mturk/tmp/heroku_server_440f97a1-gandraw2/dev/components/built_custom_components
> webpack --mode development

Hash: bd3db02f21a79fa5f060
Version: webpack 4.40.2
Time: 1081ms
Built at: 09/17/2019 12:10:28 PM
     Asset     Size  Chunks             Chunk Names
custom.jsx  346 KiB    main  [emitted]  main
Entrypoint main = custom.jsx
[react] external "react" 42 bytes {main} [built]
    + 2 hidden modules

ERROR in ./node_modules/react-sketch/dist/index.js
Module not found: Error: Can't resolve 'jsdom' in '/Users/josharnold/Desktop/desktop/paper2/ParlAI/parlai/mturk/tmp/heroku_server_440f97a1-gandraw2/dev/components/built_custom_components/node_modules/react-sketch/dist'
 @ ./node_modules/react-sketch/dist/index.js 1:85-101
 @ ./components/custom.jsx
npm ERR! code ELIFECYCLE
npm ERR! errno 2
npm ERR! custom_built_frontend@1.0.0 dev: `webpack --mode development`
npm ERR! Exit status 2
npm ERR! 
npm ERR! Failed at the custom_built_frontend@1.0.0 dev script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/josharnold/.npm/_logs/2019-09-17T19_10_28_268Z-debug.log
Traceback (most recent call last):
  File "run.py", line 143, in <module>
    main()
  File "run.py", line 45, in main
    task_directory_path=os.path.dirname(os.path.abspath(__file__))
  File "/Users/josharnold/Desktop/desktop/paper2/ParlAI/parlai/mturk/core/mturk_manager.py", line 1080, in setup_server
    tmp_dir=self.opt['tmp_dir'],
  File "/Users/josharnold/Desktop/desktop/paper2/ParlAI/parlai/mturk/core/server_utils.py", line 609, in setup_server
    tmp_dir=tmp_dir,
  File "/Users/josharnold/Desktop/desktop/paper2/ParlAI/parlai/mturk/core/server_utils.py", line 308, in setup_heroku_server
    'Webpack appears to have failed to build your '
Exception: Webpack appears to have failed to build your custom components. See the above for more info.
Joshs-MacBook-Pro:gan_draw_2 josharnold$ 

Lastly, here is a zipped folder of my task for inspection:

gan_draw_2.zip

[JackUrb]

Seems this particular issue is something with react-sketch itself: https://github.com/tbolis/react-sketch/issues/66, with a solution in https://github.com/tbolis/react-sketch/pull/57 that hasn't been pushed yet.

[josharnoldjosh]

I fixed it! I had to do npm install fabric --save, and then drag and drop all the .js and .jsx files and do a import ./file_name.js! thank you for all the help!

[josharnoldjosh]

@pringshia by any chance, does this work for the latest version of react material ui which I know uses react 16.8.0 including react hooks? I tried the above steps and still received the same error?

[pringshia]

@josharnoldjosh - Taking a look...

[pringshia]

@josharnoldjosh After following @JackUrb's instructions on your zipped code, I'm not getting any errors when doing npm install react-material-ui --save and then npm run dev.

Can you please share the contents of your webpack.config.js file?

[josharnoldjosh]

@pringshia, sure I will look for it now. But also, I didn't get errors installing or importing, but if you add a component that actually uses the library in custom.jsx such as the Card:

class Test extends React.Component {
    render() {
        return <Card> Hello world </Card>;
    }
}

Thats when I get the error. Has this happened to you with react material ui version 4?

[pringshia]

@josharnoldjosh

I'm using v4.4.2. Are you getting build errors or runtime errors in the browser?

I just added:

import { Card } from "@material-ui/core";

and

class UIContainer extends React.Component {
  render() {
    return <Card>Test</Card>;
  }
}

and am not getting any build errors.

[josharnoldjosh]

Really? When I do run.py the entire UI goes white and I get react-hook errors in the HTML console/Google Chrome console? Does this happen for you? But there are no actual errors in the terminal? @pringshia

[josharnoldjosh]

Also @pringshia I believe the install for material ui through npm is: npm install @material-ui/core --save instead of npm install react-material-ui --save

[josharnoldjosh]

From doing something like:

import { Card } from "@material-ui/core";

class UIContainer extends React.Component {
  render() {
    return <Card>Test</Card>;
  }
}

var LeftPaneHolder = {
  Evaluator: UIContainer,
  'Onboarding Evaluator': UIContainer,
  Answerer: UIContainer,
  'Onboarding Answerer': UIContainer,
}

When I run the task I get this:

With these errors:

[pringshia]

Got it. I had only checked for build errors. I’ll take a look at runtime errors and get back to you.

On Sep 20, 2019, at 1:50 PM, Josh Arnold notifications@github.com wrote:

From doing something like:

import { Card } from "@material-ui/core";

class UIContainer extends React.Component { render() { return Test; } }

var LeftPaneHolder = { Evaluator: UIContainer, 'Onboarding Evaluator': UIContainer, Answerer: UIContainer, 'Onboarding Answerer': UIContainer, } When I run the task I get this:

With these errors:

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[pringshia]

Hey @josharnoldjosh. I have a potential solution for you and also some additional context:

So the issue is that the latest version of Material UI uses React Hooks (and thus expects a newer version of React as a dependency). Our codebase has not been yet modernized to the latest version, I had started initial work (as mentioned here) but this has not yet landed in the codebase.

The easiest & fastest way to get you unblocked here is to recommend you to downgrade your dependency on @material-ui/core to version 3.x. You can do so by running the following in your frontend/ folder: npm install --save @material-ui/core@3 @material-ui/icons@3. Please let me know if this works. The downside here of course is that you'll have to use an older version of material-ui, but this should get you unblocked for now.

[josharnoldjosh]

Thanks @pringshia. We can we expect Parl.ai to be upgraded to latest verison of react? Thanks!

[pringshia]

@josharnoldjosh We're currently working on a project that's a complete overhaul of this workflow, so we haven't prioritized incremental updates

[github-actions[bot]]

This issue has not had activity in 30 days. Marking as stale.