Questions/comments about the code

[xiulinyang]

Hi,

Thank you for providing this useful dataset. I tried to run your model on another dataset but I met some difficulties. I also have some questions regarding the code. It would be great if you can help me out. :)

(1) In the predict.py file, it says import coref_model as cm, but there is no such file named like this. There was one called anaphora_model.py but there is no such method as get_predicted_antecedents. I'm not sure if you forgot to upload the coref_model.py or the other existing file is not complete yet.

(2) When I tried to run setup_all.sh, it gave me an error message saying no such file or directory: 'coref_kernels.cc'. May I know how I can deal with that?

Also, the tensorflow version is not available anymore. I would really appreciate it if you could provide some suggestions regarding the questions. I hope to hear from you soon. :)

Many thanks again for your time and kind help.

[biaoyanf]

Hi, @xiulinyang,

For the questions, (1) As stated in README.md (evaluation section), please use evaluate_folds.py for evaluation. The predict.py is the old file I used and it is of no use to this repository. Thanks for pointing this out.

(2) Might this be the issue with the TensorFlow version? Also, have you tried the other command g++ -std=c++11 -shared coref_kernels.cc -o coref_kernels.so -fPIC ${TF_CFLAGS[@]} ${TF_LFLAGS[@]} -O2?

And for the TensorFlow version, I have no issues with downloading tensorflow-gpu==1.12, what is your question? And actually, tf 1.14 or 1.15 works too.

Hope it helps.

[chanind]

It looks like the setup_all.sh is similar to https://github.com/mandarjoshi90/coref or https://github.com/kentonl/e2e-coref, but in both those repos there's an explicit coref_kernels.cc file in the repo, which seems like it was not included in this repo. In both repos this file is identical. Was this an accidental oversight? Where does the setup_all.sh code come from, is it from a tutorial? If I copy https://github.com/mandarjoshi90/coref/blob/master/coref_kernels.cc into this repo and run setup_all.sh it works.

[biaoyanf]

Thanks, @chanind. As stated in the paper, our code is originally based on the e2e coref framework from Lee et al. I have also uploaded the coref_kernels.cc to this repository.

Let me know if there is any further issue.

[chanind]

Thanks @biaoyanf! Other issues I've run into:

• the scripts also require glove_50_300_2.txt, which is not mentioned in the README and appears to come from https://drive.google.com/file/d/1fkifqZzdzsOEo0DXMzCFjiNXqsKG_cHi (mentioned in this issue: https://github.com/kentonl/e2e-coref/issues/66#issuecomment-511747025)
• the scripts require train.english.jsonlines, dev.english.jsonlines, and test.english.jsonlines, but these don't exist in the RecipeRef dataset and I can't figure out where they come from or how to create them.
• the scripts crash requiring something called elmo_cache.hdf5, which I assume is generated by the cache_elmo.py script, but this also fails as it references a hardcoded path /data/scratch/projects/punim0478/byron/Embedding/elmo/. It looks like this also needs elmo data as well? This also looks like it requires the missing jsonlines files.
• running the scripts references glove.840B.300d.txt.filtered instead of glove.840B.300d.txt, it's not clear what this is. There's a script filter_embeddings.py which looks like it can create this file but requires the above mentioned missing jsonlines files as input?

[biaoyanf]

Hi @chanind, for the issues:

• thanks for pointing it out. I have added this information to REDME.md;
• To create the jsonslines file, I wrote a script to tokenize the original text file using the nltk tool and align it with ann annotations. I have also uploaded convert_brat_into_training_format-clear.ipynb about how I processed it in case you need it.
• yes, it is from cache_elmo.py. I have modified that code where the elmo can be extracted and it should work now.
• and yes. the filtered file is fromglove.840B.300d.txt. Basically, it is used to accelerate the process. Either way, you can replace it with the original glove.840B.300d.txt too.

[rwinn1103]

Hi @biaoyanf, thanks for following up with the issues listed above. I am also attempting to reproduce your results from the paper ""What does it take to bake a cake? The RecipeRef corpus and anaphora resolution in procedural text". While attempting to setup/run this repository I have run into the following issues:

• Similar to previously mentioned, I am confused on how to acquire the required jsonlines file. When I run the provided ipynb file (which has hard coded paths within that needed to be changed), I am only output with a file named "train_converted_jsonlines_no_state" rather than train.english.jsonlines, dev.english.jsonlines, etc. Am I missing a step?
• How do I acquire the glove.840B.300d.txt.filtered file? I realize that it's not necessary, but the README doesn't include the steps needed to be taken (i.e., which scripts to run in what order) to acquire the file.
• Whenever I try to run an experiment, I run into a error of not having elmo_cache.hdf5. After your most recent update, ideally the cache_elmo.py code should extract the elmo data. However, I have run into two problems: First, the current coded link that you provide is outdated and needs to be changed to version 3 rather than version. Second, even when the version is updated (or if kept the same as is currently is at version 2), the scripts cannot extract the elmo data.

I am unsure on how to proceed given the issues listed and would appreciate some guidance if possible. Thank you.

[biaoyanf]

Hi, @rwinn1103 Thanks for your interest. For the issues:

1. Yes, it will only produce the training file. You can generate the dev and test files by changing the variable data_type = "train" to "dev" and "test". And the variable revise = True/False controls if you wanna merge the state info.
2. Thanks for pointing it out. After running setup_all.sh, you also need to run setup_training.sh. And to obtain the file you want, run the code python filter_embeddings.py ./glove.840B.300d.txt train.english.jsonlines dev.english.jsonlines in setup_training.sh
3. To produce elmo_cache.hdf5 file, run the codepython cache_elmo.py train.english.jsonlines dev.english.jsonlines test.english.jsonlines in setup_training.sh.

Let me know if you have further issues.

[rwinn1103]

Hi, @biaoyanf thank you for your response on my issues.

With this new information, I have been able to (I believe) setup the beginning of the experiments. However, I have run into a few further issues that I would very much appreciate your help with.

1. Just to note, the experiments require a "data/full/" folder that contains all of the annotations in the utilized dataset. This was not mentioned and it may prove beneficial to include it in the README when formatting one's data.
2. When running the experiments, I see that every 1000 steps the model calculates the Precision, Recall, and F1 score. However, I am running into the following error:

"Error: Could not find or load main class au.com.nicta.csp.brateval.CompareEntities get None type error in mention_results_match with label : transformed return lable: transformed with (0, 0, 0)"

This error is raised for every type of bridging (i..e, transformed, ingredient without state change, ingredient with state change). Thus, I am continually receiving metrics of simply 0 at every 1000 steps:

"mention_results (Precision, Recall and F1): {'transformed': (0, 0, 0), 'ingredient_without-state-change_associated': (0, 0, 0), 'ingredient_with-state-change_associated': (0, 0, 0), 'all_bridging': (0, 0, 0), 'all': (0, 0, 0)}"

Am I missing an additional set up step in the process of running an experiment (e.g., bridging) or do you have an idea on how to fix this issue? Thanks in advance for the guidance.

[biaoyanf]

HI, @rwinn1103 Thanks for pointing it out. For the second issue, I talked to my colleagues. It seems that they have updated the brateval tool and changed the input and output format - That's why causing the issue. If you wish to use the updated brateval, please change the cmds in brat.py for extracting the brateval evaluation. Alternatively, I have uploaded the brateval version I used for this paper - That should solve the issue.