Henkel-code-vectors

[obaysal]

https://github.com/researchart/fse18/tree/master/submissions/code-vectors

[issueFinder]

@jjhenkel I'm having problem running end to end demo on my OS X El Capitan (10.11.6). When executing "make end-to-end-redis" command, it seems some of the binary files fail to execute:

git submodule init
git submodule update --remote
[code-vectors] Ensuring we have lsee
docker pull jjhenkel/lsee
Using default tag: latest
latest: Pulling from jjhenkel/lsee
Digest: sha256:a637f271ea6d19f5485800e850e1d03ec4618fb9ea88dc6fece0e9e35d922b37
Status: Image is up to date for jjhenkel/lsee:latest
[code-vectors] Ensuring that we have c2ocaml
docker pull jjhenkel/c2ocaml
Using default tag: latest
latest: Pulling from jjhenkel/c2ocaml
Digest: sha256:47f25b23e0112707ccfa6745d9d9cba9c3d4578be83e55d9d7d84afd5d551153
Status: Image is up to date for jjhenkel/c2ocaml:latest
[code-vectors] Running end-to-end pipeline on redis...
[code-vectors] Transforming sources...
pushd /Users/issuefinder/Desktop/FSE/vector/code-vectors-artifact/c2ocaml ; /Users/issuefinder/Desktop/FSE/vector/code-vectors-artifact/tools/make redis --output-sync ; popd
~/Desktop/FSE/vector/code-vectors-artifact/c2ocaml ~/Desktop/FSE/vector/code-vectors-artifact
/bin/bash: /Users/issuefinder/Desktop/FSE/vector/code-vectors-artifact/tools/make: cannot execute binary file
~/Desktop/FSE/vector/code-vectors-artifact
[code-vectors] Generating traces...
pushd /Users/issuefinder/Desktop/FSE/vector/code-vectors-artifact/lsee ; /Users/issuefinder/Desktop/FSE/vector/code-vectors-artifact/tools/make redis --output-sync ; popd
~/Desktop/FSE/vector/code-vectors-artifact/lsee ~/Desktop/FSE/vector/code-vectors-artifact
/bin/bash: /Users/issuefinder/Desktop/FSE/vector/code-vectors-artifact/tools/make: cannot execute binary file
~/Desktop/FSE/vector/code-vectors-artifact
[code-vectors] Collecting traces...
pushd /Users/issuefinder/Desktop/FSE/vector/code-vectors-artifact/lsee ; NAME=redis /Users/issuefinder/Desktop/FSE/vector/code-vectors-artifact/tools/make collect ; popd
~/Desktop/FSE/vector/code-vectors-artifact/lsee ~/Desktop/FSE/vector/code-vectors-artifact
/bin/bash: /Users/issuefinder/Desktop/FSE/vector/code-vectors-artifact/tools/make: cannot execute binary file
~/Desktop/FSE/vector/code-vectors-artifact
[code-vectors] Completed end-to-end run on redis!
[code-vectors] Run make learn-vectors-redis to learn vectors using GloVe!

I believe this is a problem, because the command related to learning vectors terminates with the following error:

[code-vectors] Ensuring we have GloVe
docker pull jjhenkel/glove
Using default tag: latest
latest: Pulling from jjhenkel/glove
Digest: sha256:41b963b19ae421773164f4280c1a47f7e984377e9e5af58055dcf3f9e23bb2a3
Status: Image is up to date for jjhenkel/glove:latest
[code-vectors] Learning vectors for traces generated from redis...
docker run -it --rm \
      -v /Users/issuefinder/Desktop/FSE/vector/code-vectors-artifact/lsee:/traces \
        -v /Users/issuefinder/Desktop/FSE/vector/code-vectors-artifact/artifacts/redis:/output \
        jjhenkel/glove \
        /traces/redis.traces.txt 10 15 300 50
[GloVe] Learning vectors with the following parameters:
[GloVe]               corpus: /traces/redis.traces.txt
[GloVe]      vocab min count: 10
[GloVe]     vector dimension: 300
[GloVe]          window size: 15 (window is symettric)
[GloVe]  max # of iterations: 50
[GloVe]         # of threads: 4
[GloVe]         memory limit: 4.0
[GloVe]            verbosity: 2
[GloVe]          file format: 0
[GloVe]                X max: 10
[GloVe] Building vocabulary...
/app/entrypoint.sh: line 43: /traces/redis.traces.txt: No such file or directory
make: *** [learn-vectors-redis] Error 1

Could you please help me resolve the issue?

[jjhenkel]

Hi @issueFinder ,

Thank you for the detailed response and logs. I've pushed a fix to the repo/submodule that should remove this problem. Could you please run the following commands:

git clone https://github.com/jjhenkel/code-vectors-artifact.git
cd ./code-vectors-artifact
make end-to-end-redis
make learn-vectors-redis

If you still run into issues please let me know---as I mentioned in the INSTALL.md we've only done limited testing on OSX; but, at the very least, our Docker images for reproducing the experiments should have no trouble running on any platform with Docker installed.

Thanks!

[issueFinder]

@jjhenkel

Now that "make end-to-end-redis" works, I faced the following error while executing "make learn-vectors-redis" command:

[code-vectors] Ensuring we have GloVe
docker pull jjhenkel/glove
Using default tag: latest
latest: Pulling from jjhenkel/glove
Digest: sha256:41b963b19ae421773164f4280c1a47f7e984377e9e5af58055dcf3f9e23bb2a3
Status: Image is up to date for jjhenkel/glove:latest
[code-vectors] Learning vectors for traces generated from redis...
docker run -it --rm \
      -v /Users/issuefinder/Desktop/FSE/vector/code-vectors-artifact/lsee:/traces \
        -v /Users/issuefinder/Desktop/FSE/vector/code-vectors-artifact/artifacts/redis:/output \
        jjhenkel/glove \
        /traces/redis.traces.txt 10 15 300 50
[GloVe] Learning vectors with the following parameters:
[GloVe]               corpus: /traces/redis.traces.txt
[GloVe]      vocab min count: 10
[GloVe]     vector dimension: 300
[GloVe]          window size: 15 (window is symettric)
[GloVe]  max # of iterations: 50
[GloVe]         # of threads: 4
[GloVe]         memory limit: 4.0
[GloVe]            verbosity: 2
[GloVe]          file format: 0
[GloVe]                X max: 10
[GloVe] Building vocabulary...
BUILDING VOCABULARY
Processed 34204876 tokens.
Counted 7532 unique words.
Truncating vocabulary at min count 10.
Using vocabulary of size 4109.

[GloVe] Building cooccurence matrix...
/app/entrypoint.sh: line 55:    10 Illegal instruction     $BUILDDIR/cooccur -memory $MEMORY -vocab-file $VOCAB_FILE -verbose $VERBOSE -window-size $WINDOW_SIZE < $CORPUS > $COOCCURRENCE_FILE
make: *** [learn-vectors-redis] Error 132

Could you please fix this one? Thanks.

[jjhenkel]

Hi @issueFinder ,

Apologies for the second inconvenience---this one seems to be because GloVe was compiled against our CentOS workstation as part of the build-process for that docker image (and, because of this, may be using some unsupported instructions).

I've re-built that image to make it re-compile GloVe before it runs. It's been pushed to Docker Hub.

Could you try just re-running make learn-vectors-redis now? It should pull the new image. No re-clone of the repo necessary.

Thanks!

[issueFinder]

@jjhenkel a new issue came up:

[code-vectors] 5 closest words to "clusterAddSlot"
Traceback (most recent call last):
  File "/app/redis-demo.py", line 26, in <module>
    demo('clusterAddSlot')
  File "/app/redis-demo.py", line 11, in demo
    for (i, (n,_)) in enumerate(kv.most_similar(word, topn=5)):
  File "/usr/local/lib/python3.6/site-packages/gensim/models/keyedvectors.py", line 365, in most_similar
    mean.append(weight * self.word_vec(word, use_norm=True))
  File "/usr/local/lib/python3.6/site-packages/gensim/models/keyedvectors.py", line 274, in word_vec
    raise KeyError("word '%s' not in vocabulary" % word)
KeyError: "word 'clusterAddSlot' not in vocabulary"
make: *** [learn-vectors-redis] Error 1

[jjhenkel]

Hi @issueFinder ,

At this point in the process the following things have happened:

1. Redis has been compiled and the source-to-source transformation has occurred.

2. Lightweight symbolic traces have been generated from the transformed sources.

3. GloVe has learned vectors using this trace corpus and saved them to disk.

The step that is failing is the very last step: a demo script I wrote to probe the learned vectors.

Sadly, I'm not able to reproduce this on my Ubuntu desktop or the CentOS workstation that ran the original experiments. So, at this point, given that everything has run except this demo script, I'd say it's probably best to move on to the experiment reproductions---they are 'smaller' in that they don't require orchestrating this complex workflow on an unknown machine.

Thank you for your patience and persistence. An alternative would be to try the end-to-end on a non-OSX machine.

[issueFinder]

@jjhenkel Can you modify the script so that it ignores vocabularies that are not found in the dictionary? It might probably impact the accuracy of the result, but may work on having an end to end demo of the tool.

[jjhenkel]

@issueFinder I've prepared an alternate demo script that avoids any assumptions about what words may or may not be in the vocabulary GloVe uses. Here's the example output I get from a run I just completed:

[jjhenkel@velveeta] (23)$ docker run -it --rm -v `pwd`/artifacts:/artifacts --entrypoint python jjhenkel/glove /app/redis-demo.py
Ten most similar word pairs:
  - word "geohash_move_x" is closest to word "geohash_move_y"
  - word "geohash_move_y" is closest to word "geohash_move_x"
  - word "je_bin_postfork_parent" is closest to word "je_extents_postfork_parent"
  - word "je_extents_postfork_parent" is closest to word "je_bin_postfork_parent"
  - word "bin_postfork_parent" is closest to word "extents_postfork_parent"
  - word "extents_postfork_parent" is closest to word "bin_postfork_parent"
  - word "?(&->ht[1])->sizemask" is closest to word "?(&->ht[0])->sizemask"
  - word "?(&->ht[0])->sizemask" is closest to word "?(&->ht[1])->sizemask"
  - word "je_arena_dalloc_bin_junked_locked" is closest to word "arena_dalloc_bin_junked_locked"
  - word "arena_dalloc_bin_junked_locked" is closest to word "je_arena_dalloc_bin_junked_locked"

You shouldn't need to re-clone the repository. Just run make learn-vectors-redis again. If this doesn't work we can try debugging using the output from GloVe and/or the learned vectors stored in ./artifacts/redis/vectors.txt.

[issueFinder]

This artifact is well-documented and instructions are easy to follow. I was able to regenerate the evaluation results as well as executing an end to end demo of the tool using the installation instructions and fixes that they made. Hence, the artifact is "Functional".

Authors have made their tool publicly available on different Github repositories and a publicly available repository with a DOI link. Thereby, I believe the artifact can get the "Available" badge as requested by the authors.

[jjhenkel]

Hi,

@timm I noticed this issue was closed? Was there supposed to be a second review of this artifact? I was holding off on comments until the second review. That being said, I have two questions:

1. @issueFinder would you consider this artifact well-documented enough to be Reusable?

2. Given that @issueFinder was able to 'regenerate the evaluation results' (our tables and figures for all of our research questions) could we consider this artifact Replicated?

Many thanks to both @timm and @obaysal for organizing and @issueFinder for their diligent review.

[timm]

Many thanks to both @timm and @obaysal for organizing and @issueFinder for their diligent review.

[timm]

issue was closed july21 when we made our final determination.

for papers with <2 reviews, the chairs reviewed and made a determination