search

researchart / rose6icse

12 stars 71 forks source link

weimerCodeSynthViewer #7

Closed timm closed 4 years ago

timm commented 4 years ago

https://github.com/researchart/rose6icse/tree/master/submissions/available/weimerCodeSynthViewer

Below are a list of emails and github ids for the authors of Submission #143, "Neurological Divide: An fMRI Study of Prose and Code Writing":

@rkruegs123 <=== corresponding author @iamyhhy @xinyuliu12 @tsantander @weimer @kjl2y

applying for available and reusable

timm commented 4 years ago

@rkruegs123

minkull commented 4 years ago

Author

\author{Cheng Wen}
\affiliation{
  \institution{CSSE, Shenzhen University}
  \city{Shenzhen}
  \country{China}}

\author{Haijun Wang}
\affiliation{
  \institution{CSSE, Shenzhen University}
  \city{Shenzhen}
  \country{China}}

\author{Yuekang Li}
\affiliation{
  \institution{Nanyang Technological University}
  \country{Singapore}}

\author{Shengchao Qin}
\affiliation{
  \institution{Teesside University}
  \city{Tees Vally}
  \country{UK}}

\author{Yang Liu}
\affiliation{
  \institution{Nanyang Technological University}
  \country{Singapore}}

\author{Zhiwu Xu}
\affiliation{
  \institution{CSSE, Shenzhen University}
  \city{Shenzhen}
  \country{China}}

\author{Hongxu Chen}
\affiliation{
  \institution{Nanyang Technological University}
  \country{Singapore}}

\author{Xiaofei Xie}
\affiliation{
  \institution{Nanyang Technological University}
  \country{Singapore}}

\author{Geguang Pu}
\affiliation{
  \institution{East China Normal University}
  \city{Shanghai}
  \country{China}}

\author{Ting Liu}
\affiliation{
  \institution{Xi'an Jiaotong University}
  \city{Xi'an}
  \country{China}}
ICSE20ArtifactAnonymousReviewer commented 4 years ago

The authors apply for the badges reusable and available.

*Reusable: The code is commented and there is a good documentation of how to use the system. For reusing and re-purposing of the code, it would be good to also add a short description (maybe model) to provide an overview of the systems architecture.

License: GNU General Public License

Functional: I did get the system to run without errors being thrown after installing all described packages. Unfortunately, the system does not return from there to allow the user to type. This means that "open presenter.html" and "=" were not taken as input. Also manually opening presenter.html, did not work. I tried this using the commands: "node server.js 99 0", "node server.js 99 1", and "node server.js 99 2", with no difference. Is there a hidden step that was not described?

*Available There is only example data on the Github. The real data is on another university server. Is that data meant to be part of the artifact? Furthermore, there is no DOI delivered with the artifact. I propose to use an archiving service comparable to Zendo (https://zenodo.org/, https://guides.github.com/activities/citable-code/).

ghost commented 4 years ago

Functional: I had the same problem as the other reviewer. However, in my case, I am not sure that I was able to successfully install the system. The artefact should contain a step-by-step installation guide (kind of installation for dummies). I will comment on the issues I found, and the steps I followed. 1.- I download the artefact from GitHub. 2.- I install node-v12.16.0-pkg for Mac (since it is the recommended, but not last, version) 3.- I add /usr/local/bin to $PATH. I notice about this when I install the node package. The installation guide the authors provide does not mention this and should. 4.- I start the backend server, but does not work. I continue reading, and see the prerequisites. They appear too late. The readme file should be better organised. 5.- I install the required packages. But I am not sure that I have succeed. I start with http, and get the following output:

npm WARN saveError ENOENT: no such file or directory, open '/Users/…/ICSE Artifacts 2020/code-synth-viewer/package.json'
npm WARN enoent ENOENT: no such file or directory, open '/Users/…/ICSE Artifacts 2020/code-synth-viewer/package.json'
npm WARN code-synth-viewer No description
npm WARN code-synth-viewer No repository field.
npm WARN code-synth-viewer No README data
npm WARN code-synth-viewer No license field.

+ http@0.0.0
updated 1 package and audited 480 packages in 2.248s
found 0 vulnerabilities

Same errors with the rest of the packages. Then I notice that I do not have a file named package.json, but package-lock.json. I rename package-lock.json as package.json, and try to reinstall. For http, I get the following output:

npm install http
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN code-synth-viewer No repository field.
npm WARN code-synth-viewer No license field.

+ path@0.12.7
added 4 packages from 2 contributors and audited 5 packages in 0.661s
found 0 vulnerabilities

I do not understand why I should commit the file. I must be missing something.

When installing the rest of the packages, I get the same type of result, varying the message obtained in the number of packages added and audited and the number of contributors.

I run node server.js 99 0 with the same result as the other reviewer. I proceed replacing 0 with 1, 2 and 3, and the same results.

Reusable: Since I have not been able to run the application, I think the documentation of the artefact needs to be improved.

Available: There is a public link to the artefact (can be downloaded from Github). However, the link to

all participant keystroke and answer data

does not seem to work.

timm commented 4 years ago

@rkruegs123
@iamyhhy @xinyuliu12 @tsantander @weimer @kjl2y

please jump in and help reviewers get your artifact running. they seem eager to help

rkruegs123 commented 4 years ago

Thank you both for your replies. There was indeed a missing step in the installation that you both seem to have encountered. Below is an updated set of installation instructions (it is updated on the README as well)

  1. Download/clone the repository
  2. Verify that nodejs is installed (e.g., node -v). If it is not, install it.
  3. Navigate to the code-synth-viewer directory (i.e., cd path/to/code-synth-viewer)
  4. Run npm init to create a package.json file (accept all defaults)
  5. While staying in the code-synth-viewer directory, proceed to install all required packages with npm
  6. You should then be able to run the software normally. For example, start the server with node server.js 99 0 and open presenter.html in a web browser other than Firefox, and hit the '=' key.

Regarding data availability, the link to all keystroke and answers data should now be working. While the artifact is the software, and not the data, we make these data available for reproducibility purposes.

Thank you again for your feedback, and please let me know if this resolves the issue.

ICSE20ArtifactAnonymousReviewer commented 4 years ago

Dear @rkruegs123,

Thanks for the new instructions!

node -v leads to the following: v8.11.4

Then node init leads to an error: \code-synth-viewer>node init module.js:549 throw err; ^ Error: Cannot find module '...\code-synth-viewer\init' at Function.Module._resolveFilename (module.js:547:15) at Function.Module._load (module.js:474:25) at Function.Module.runMain (module.js:693:10) at startup (bootstrap_node.js:191:16) at bootstrap_node.js:612:3

I tried that on two cases:

For the old clone, node server.js 99 0 leads to the same result as before. I opened presenter.html in the internet explorer and pressed "=". However, with my keyboard layout I have to "=" key as such, but have to press shift and 0. Maybe that causes the problem?

rkruegs123 commented 4 years ago

Can you try npm init instead of node init? I have updated my previous comment to reflect this.

ICSE20ArtifactAnonymousReviewer commented 4 years ago

Hi, Just tried it, and it works. However, I still cannot get any reaction to pressing "=". Here is the complete track of what I just tried:

[...]>git clone https://github.com/rkruegs123/code-synth-viewer.git Cloning into 'code-synth-viewer'... remote: Enumerating objects: 2208, done. remote: Counting objects: 100% (2208/2208), done. remote: Compressing objects: 100% (1824/1824), done. remote: Total 2208 (delta 364), reused 2157 (delta 313), pack-reused 0R Receiving objects: 100% (2208/2208), 5.58 MiB | 1.03 MiB/s, done.

Resolving deltas: 100% (364/364), done. Updating files: 100% (585/585), done.

[...]>cd code-synth-viewer

[...]\code-synth-viewer>node -v v8.11.4

[...]\code-synth-viewer>npm init This utility will walk you through creating a package.json file. It only covers the most common items, and tries to guess sensible defaults.

See npm help json for definitive documentation on these fields and exactly what they do.

Use npm install <pkg> afterwards to install a package and save it as a dependency in the package.json file.

Press ^C at any time to quit. package name: (code-synth-viewer) version: (1.0.0) description: entry point: (server.js) test command: git repository: (https://github.com/rkruegs123/code-synth-viewer.git) keywords: author: license: (ISC) About to write to [...]\code-synth-viewer\package.json:

{ "name": "code-synth-viewer", "version": "1.0.0", "description": "The software used for conducting experiments as described in \"Neurological Divide: An fMRI Study of Prose and Code Writing\" at ICSE 2020. This repository permits both (1) the reproduction and validation of our results and experimental protocol, as well as (2) the adaptation of this software for similar human studies of software engineering activities (with or without medical imaging). For reproduction of the conclusions drawn from the data collected in our study, we make available all participant keystroke and answer data, as well as the de-identified medical imaging data (contact Dr. Westley Weimer).", "main": "server.js", "scripts": { "test": "echo \"Error: no test specified\" && exit 1", "start": "node server.js" }, "repository": { "type": "git", "url": "git+https://github.com/rkruegs123/code-synth-viewer.git" }, "author": "", "license": "ISC", "bugs": { "url": "https://github.com/rkruegs123/code-synth-viewer/issues" }, "homepage": "https://github.com/rkruegs123/code-synth-viewer#readme" }

Is this OK? (yes)

[...]\code-synth-viewer>npm install shuffle-seed npm notice created a lockfile as package-lock.json. You should commit this file.

[...]\code-synth-viewer>npm install http

[...]\code-synth-viewer>npm install path

[...]\code-synth-viewer>npm install request npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142

[...]\code-synth-viewer>npm install express

[...]\code-synth-viewer>npm install fs

[...]\code-synth-viewer>node server.js 99 0

rkruegs123 commented 4 years ago

It sounds like all is working up to your node server.js 99 0. Allow me to be more explicit with this step:

  1. Install npm packages as you did...
  2. node server.js 99 0
  3. Open presenter.html in a web browser other than Firefox
  4. Press = in the web browser not the terminal
  5. Upon the key press, you should see output in the terminal indicative of the server starting. However, it will take ~20-30 seconds until you see text appear on the web browser.

Can you confirm that this is what you tried?

ICSE20ArtifactAnonymousReviewer commented 4 years ago

Yes, I can confirm that this is what I did.

rkruegs123 commented 4 years ago

It is likely an issue with the Javascript keycode we use to recognize = in our environment. Can you tell me what OS, browser, and browser version you are using? I can then either change the keycode to match your environment, or suggest a browser to use with the current setup.

Once we get this working, I will add some more details re. this to the README. Thank you for bearing with us here!

ICSE20ArtifactAnonymousReviewer commented 4 years ago

Sure, I use Windows (10) with Microsoft Edge 42.17134......0.

rkruegs123 commented 4 years ago

Okay. Could you open this link (https://keycode.info/) and enter = as you would, and tell me the number that is shown? If the number is not 187 this is likely the issue.

You can then (1) report the number to me and I will add support for Microsoft Edge accordingly, and/or (2) try a different browser (e.g., Chrome)

ICSE20ArtifactAnonymousReviewer commented 4 years ago

The main number is 48. There is no difference between pressing 0 or shift-0, which is what I need to press to get "=". However, the event.key detects the "=" on Edge. (By the way, I get the completely same results on Firefox.)

Can you add support for Edge?

rkruegs123 commented 4 years ago

Thank you for checking!

I have temporarily changed the keycode to 48. Once you confirm that this was the problem, I will update the README.

timm commented 4 years ago

@ICSE20ArtifactAnonymousReviewer @anonymousICSEartifacts

I assert that "available" does not require executable products, but "reusable" does

"reusable" needs running code and some DOI link (eg. from Zenodo) to artifacts stored in some permanent storage facility (GitHub)

can we at least certify this one as "available"?

rkruegs123 commented 4 years ago

I have created a DOI link for the artifact on Zenodo: https://doi.org/10.5281/zenodo.3674328

@ICSE20ArtifactAnonymousReviewer @anonymousICSEartifacts Please let me know if you have further trouble getting the software to work -- once you have setup the software with the npm init addition I provided, I am confident that the software will function for you provided that the appropriate keycode is set for your browser. It is currently set to 48, per the above conversation. To support a different browser/OS combo, you can either let me know and I will update repo for you or you can change the value yourself (as would be required by a user of the software). This would be done in the following way:

ICSE20ArtifactAnonymousReviewer commented 4 years ago

Hi, I finally can confirm that it works! Note that I changed to internet explorer in the end, as there was also an issue with allowing the script to run. That might be a note to add to the installation guide for trouble-shooting.

Thanks for all the help and cooperation!

rkruegs123 commented 4 years ago

Ah, that is helpful to know -- I will be sure to update the README with that information, as well as other things we discovered in this thread. Thanks for working with me on this!

ICSE20ArtifactAnonymousReviewer commented 4 years ago

@timm I think we can certify both, available and reusable, now.

ghost commented 4 years ago

I have followed the new instructions, and this is what I get. I am running the software in a MacBook Pro with Mojave:

node -v
v12.16.0

I proceed with npm init:

This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sensible defaults.

See `npm help json` for definitive documentation on these fields
and exactly what they do.

Use `npm install <pkg>` afterwards to install a package and
save it as a dependency in the package.json file.

Press ^C at any time to quit.
package name: (code-synth-viewer) 
version: (1.0.0) 
git repository: 
keywords: 
author: 
license: (ISC) 
About to write to /Users/.../ICSE Artifacts 2020/code-synth-viewer/package.json:

{
  "requires": true,
  "lockfileVersion": 1,
  "dependencies": {
    "express": "^4.17.1",
    "fs": "0.0.1-security",
    "http": "0.0.0",
    "path": "^0.12.7",
    "request": "^2.88.2",
    "shuffle-seed": "^1.1.6"
  },
  "scripts": {
    "start": "node server.js"
  },
  "name": "code-synth-viewer",
  "description": "The software used for conducting experiments as described in \"Neurological Divide: An fMRI Study of Prose and Code Writing\" at [ICSE 2020](https://conf.researchr.org/home/icse-2020). This repository permits both (1) the reproduction and validation of our results and experimental protocol, as well as (2) the adaptation of this software for similar human studies of software engineering activities (with or without medical imaging). For reproduction of the conclusions drawn from the data collected in our study, we make available [all participant keystroke and answer data](https://dijkstra.eecs.umich.edu/fmri/icse20-data), as well as the de-identified medical imaging data (contact [Dr. Westley Weimer](weimerw@umich.edu)).",
  "version": "1.0.0",
  "main": "server.js",
  "devDependencies": {},
  "author": "",
  "license": "ISC"
}

Is this OK? (yes) yes

Then I proceed with installation of needed packages, but get always the same error:

npm install http
npm WARN code-synth-viewer@1.0.0 No repository field.

I look at the output obtained by the other reviewer when running "npm init" and notice that in my case the repository field appears empty. Could this be because I downloaded the repository instead of cloning it?? Please, keep this in mind for your installation instructions. Also, It took me a while to notice that I had to press "enter" with a new line in "npm init". Please, also take that into consideration.

I re-run npm init providing this time the name of the repository and then install packages. This time it seems to work. This is the obtained output:

npm install http
npm notice created a lockfile as package-lock.json. You should commit this file.
+ http@0.0.0
added 1 package and audited 1 package in 1.175s
found 0 vulnerabilities

npm install path
+ path@0.12.7
added 4 packages from 2 contributors and audited 5 packages in 0.657s
found 0 vulnerabilities

npm install request
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
+ request@2.88.2
added 47 packages from 58 contributors and audited 68 packages in 2.077s
found 0 vulnerabilities

npm install express
+ express@4.17.1
added 48 packages from 35 contributors and audited 194 packages in 1.883s
found 0 vulnerabilities

npm install fs
+ fs@0.0.1-security
added 1 package in 10.748s

npm install shuffle-seed
+ shuffle-seed@1.1.6
added 2 packages from 2 contributors and audited 197 packages in 3.128s
found 0 vulnerabilities

I type "node server.js 99 0" and open "presenter.html" with Safari. I get a window with the red "+". I type "=" in the browser and nothing happens. Does this mean Safari does not work either? Then, what browser should I use? I am asking that because I have installed in my laptop Safari and Firefox.

Another issue. I have not committed package-lock.json as stated when installing http, as I have not cloned the repository (just downloaded). Is this relevant??

I have checked and "=" is 187 in my case.

ghost commented 4 years ago

Got it!!!! I hadn´t noticed that you had temporarily changed "=" to 48 for the other reviewer (and I downloaded again the repository when starting). I edited the "presenter.html" file and restored to value of "=" back to 187. Now it works.

ghost commented 4 years ago

@timm I also think we can certify available and reusable.

ghost commented 4 years ago

Of course, provided the authors incorporate to the instructions what I have commented.

timm commented 4 years ago

@rkruegs123 @iamyhhy @xinyuliu12 @tsantander @weimer @kjl2y

we won't badge you until you address @anonymousICSEartifacts' concerns. if you've sent a pull request please advise

rkruegs123 commented 4 years ago

I have issued the corresponding pull request.

timm commented 4 years ago

@anonymousICSEartifacts

does this update satisfy your issues? what badge (if any) can you now recommend?

ghost commented 4 years ago

Hi,

The authors have not yet addressed one of my concerns. Please, read all my comments above carefully. When running "npm init" the repository field default value appears empty. I had trouble with this and only noticed about it when I saw the output of the other reviewer. I do not know if this could be because I did not clone the repository but downloaded it. Users must be warned about this.

Also, I have gone through the installation description and seems to me not clear enough. It would help if the installation steps are provided one by one, in an algorithm fashion. Something like:

  1. Install Node.js
  2. Verify that it is installed by typing node -v in terminal window
  3. Go to the code-synth-viewer directory
  4. run npm init. Here there should be an explanation about how it works. You have to press enter to accept defaults, but there might be an issue with repository name.
  5. Etc.

Perhaps separating Prerequisites from Installation would make it easier to understand.

ghost commented 4 years ago

@timm So far, only available, until my last two concerns are addressed (about the repository field empty when I installed it and lack of clarity of installation).

rkruegs123 commented 4 years ago

@anonymousICSEartifacts I will certainly address the clarity in installation steps. However, I am wondering if your issue with npm WARN code-synth-viewer@1.0.0 No repository field. is a consequence of running npm init on a directory that you already tried to install packages on before running npm init.

Could you try with a fresh clone? If this is still an issue for you, I will add it.

rkruegs123 commented 4 years ago

Otherwise, I have submitted a pull request to address your concern re. an enumerated and more clear set of installation instructions (divided by prereqs and installation). I also included some details about default arguments with ``npm init```

timm commented 4 years ago

@anonymousICSEartifacts

please see new merged version. can you now certify available and reusable? or...???

ghost commented 4 years ago

The installation steps now are clearer.

As regards the repository issue, I deleted everything (but the installation of node, which I guess is irrelevant) and dowloaded again the repository. This is the reason why I had trouble with the "=" coding. The only difference in the installation compared to the other reviewer is that I did not clone but downloaded the repository (apart from the system, i.e., I have a MacBook instead of a Windows machine). Could you please check if this could be the issue?

If you are not able to find the issue, an alternative could be to specify in the installation this issue. I mean, say that should be checked the value of repository when running nom init, and if it is empty, add the specific string.

This would be fine for me.

rkruegs123 commented 4 years ago

I was not able to reproduce the issue, and added it to the installation instructions

timm commented 4 years ago

@anonymousICSEartifacts

please see new merged version. can you now certify available and reusable? or...???

timm commented 4 years ago

So without further input, this will be available, not reusable

rkruegs123 commented 4 years ago

@timm @anonymousICSEartifacts I believe @anonymousICSEartifacts has until the 24th to confirm that it is also reusable, yes?

timm commented 4 years ago

at this stage, I assert that

Thank you to all for your responsiveness

ghost commented 4 years ago

I agree with available and reusable.

minkull commented 4 years ago

Available and reusable awarded.