Preparing Environment For Allen-nlp.

[elonmusk-01]

Hi @ikuyamada , I am facing issues while creating environment setup for allennlp based ner and re solution. I come to know that it works only on python3.7 so I dockerized it with the requirements.txt file and then run the container. This time the package poetry was not found. I added it as an extra requirement and it starts running. For both allennlp based solutions, it errored out below: How to prepare environment for this solution?? Can I use higher versions of python for that??

[aakashb95]

I think you could try python3.8, I am using python3.8 and used the poetry command as specified for allennlp. It works.

[elonmusk-01]

@aakashb95 Did you use a virtual environment? There are some packages like TensorFlow==1.15 that required lower versions of python. There are a lot of python versions and hashes specified in the requirements.txt. How to deal with them??

[aakashb95]

Hi @elonmusk-01 You are right, it does not work with python3.8. I was in a different environment when i wrote the above comment. Apologies.

I use Ubuntu 20.04 and miniconda and have poetry installed already. I cloned this repo and navigated into it.

I reproduced the environment setup and here are the steps I followed:

cd luke

conda create -n luke_env python=3.7 

conda activate luke_env

pip --version                                                                                  
# pip 21.2.2 from /home/aakash/miniconda3/envs/luke_env/lib/python3.7/site-packages/pip (python 3.7)

pip install -r requirements.txt #this goes through smoothly without any errors or warnings on my system

## For AllenNLP:
export SEED=0;
export TRAIN_DATA_PATH="tacred/train.json";
export VALIDATION_DATA_PATH="tacred/dev.json";
export TRANSFORMERS_MODEL_NAME="studio-ousia/luke-base";

poetry run allennlp train examples_allennlp/relation_classification/configs/transformers_luke.jsonnet -s results/relation_classification/luke-base --include-package examples_allennlp -o '{"trainer": {"cuda_device": 0}}'

This runs perfectly fine for me. Not sure if it is the docker container causing you some problems.

[elonmusk-01]

@ikuyamada

[elonmusk-01]

@ikuyamada Can we have documentation that confirms with things that leads to successful results of running system??

[ikuyamada]

Hi @elonmusk-01, We used Ubuntu in our experiments and confirmed that these examples can be executed without issues. Also, it seems that the error may be due to the failure on loading JSON, since get_output_dim simply returns self.metadata["hidden_size"] (see here) and self.metadata is directly loaded from a JSON file (see here).

[elonmusk-01]

@ikuyamada On this point looks like I have all the requirements as the error is different from the one I posted at the very beginning of the conversation. Can you please suggest any solution if this is related to some path issue? Does it makes any sense to manually download the model somewhere or does it download itself?? Additionally, I am using the Ubuntu image as a base image inside the docker. It will be very helpful for everyone if there's a docker file provided for these solutions :).

[elonmusk-01]

@ikuyamada Happily I managed to solve the above error by packages version tradeoff. Now I am moving forward and want to know this assertion error at the end of fine-tuning. :)

[ikuyamada]

We are sorry but we do not provide technical support as this repository is publicized for academic research. It looks like the exception is raised in the AllenNLP library. If you think this is due to the issue of the library, please directly submit an issue to the library.