search

researchart / re19

control repo for the re19 artifacts evaluation committee
BSD 2-Clause "Simplified" License
3 stars 2 forks source link

Review of submission 87vogelsang #23

Closed timm closed 5 years ago

artifact-review commented 5 years ago

Hi, We tried to create a virtual environment for Python 3.6 as per specified in requirements.txt. However, we have encountered a number of issues in doing so as listed out below:

  1. We used the following command to install all the dependencies pip3 install -r requirements.txt. Unfortunately, it could never resolve the version of 1.10.1 for the tensorflow-gpu library. The error message reads: Could not find a version that satisfies the requirement tensorflow-gpu==1.10.1

  2. Then, we tried to create an environment taking the latest version of everything. Unfortunately, it was never successful. Probably, for mismatching installation of deep learning libraries (CUDA, cuDNN, etc.). So, please specify which version of CUDA toolkit and cuDNN library needs to be installed for the desired environment.

The configuration of the machine from which we're trying to launch this project is as follows:

jonaswinkler commented 5 years ago

Hello,

We have run the software on linux (Ubuntu 16.04.1 LTS) only, Windows might be an issue, this specific version of tensorflow-gpu might not be available on that platform.

We used CUDA 9.0 and cudNN 7.1.4.

You can try to use tensorflow instead of tensorflow-gpu. This should make it easier to verify that the software is working.

Alternatively, you can also use Theano==1.0.2 instead of tensorflow and create/alter the keras configuration file, which usually is located at HOME_DIRECTORY/.keras/keras.json:

{ "floatx": "float32", "epsilon": 1e-07, "backend": "theano", "image_data_format": "channels_last" }

We will adjust the documentation accordingly.

artifact-review commented 5 years ago

Dear Authors, Thanks for the update. I am able to run the project using Theano as backend.

Upon a thorough investigation, I found the following issues which are mostly related to documentation:

  1. Data-set has not been described. Need to specify which data file is used for what. Also, the internal organization, formatting, and structure of the data inside a file need to be specified.
  2. Similarly, please specify clearly when to use which config file.
  3. The report includes results in a raw format including lots of system information. Please specify the relevant portion of the report that needs to be extracted for insight on the performance.
  4. conf\pvm-multi-pretrained.json has a comma missing, preventing project execution due to parsing error.
  5. It is better to provide a short description of labels such as Formal_verification, Process_audit etc.

As for a badge, since a publicly accessible DOI link has been provided for the artifacts and considering the above points will be addressed, I recommend a badge of Available.

neilernst commented 5 years ago

hi @artifact-review can you please clarify - the badge is recommended upon fixing the documentation? or independently?

artifact-review commented 5 years ago

@neilernst I recommend the badge of "Available" irrespective of the documentation fixes. Sorry for the confusion.

neilernst commented 5 years ago

thanks - @andivogelsang suggest when possible you make these changes anyway, as it will probably help you just as much :)

@timm please concur on this badge or comment with other changes