Closed gchhablani closed 2 years ago
@gchhablani I am not sure. When we run prepare.sh
, we run the following python script first (https://github.com/wasiahmad/PLBART/blob/main/multilingual/data/prepare.sh#L60).
PYTHONPATH=${HOME_DIR} python process.py;
This script create files with the format [train|valid|test].LANG-en_XX.LANG]
under the data/processed
directory. It's weird that you are getting the error because the process.py
file should complain first if there is an issue.
@wasiahmad Thanks!
I will check again the order in which things are run and get back. My processed
directory is empty after download.sh
. Will see what is causing the issue.
This is what I get after completing the downloading step inside the data directory.
.
|-go
| |-valid.jsonl
| |-test.txt
| |-train.txt
| |-valid.txt
| |-test.jsonl
| |-train.jsonl
|-python
| |-valid.jsonl
| |-test.txt
| |-train.txt
| |-valid.txt
| |-test.jsonl
| |-train.jsonl
|-java
| |-valid.jsonl
| |-test.txt
| |-train.txt
| |-valid.txt
| |-test.jsonl
| |-train.jsonl
|-javascript
| |-valid.jsonl
| |-test.txt
| |-train.txt
| |-valid.txt
| |-test.jsonl
| |-train.jsonl
|-encode.py
|-ruby
| |-valid.jsonl
| |-test.txt
| |-train.txt
| |-valid.txt
| |-test.jsonl
| |-train.jsonl
|-php
| |-valid.jsonl
| |-test.txt
| |-train.txt
| |-valid.txt
| |-test.jsonl
| |-train.jsonl
|-download.sh
|-process.py
|-prepare.sh
.
├── download.sh
├── encode.py
├── go
│ ├── test.jsonl
│ ├── test.txt
│ ├── train.jsonl
│ ├── train.txt
│ ├── valid.jsonl
│ └── valid.txt
├── java
│ ├── test.jsonl
│ ├── test.txt
│ ├── train.jsonl
│ ├── train.txt
│ ├── valid.jsonl
│ └── valid.txt
├── javascript
│ ├── test.jsonl
│ ├── test.txt
│ ├── train.jsonl
│ ├── train.txt
│ ├── valid.jsonl
│ └── valid.txt
├── php
│ ├── test.jsonl
│ ├── test.txt
│ ├── train.jsonl
│ ├── train.txt
│ ├── valid.jsonl
│ └── valid.txt
├── prepare.sh
├── processed
├── process.py
├── python
│ ├── test.jsonl
│ ├── test.txt
│ ├── train.jsonl
│ ├── train.txt
│ ├── valid.jsonl
│ └── valid.txt
└── ruby
├── test.jsonl
├── test.txt
├── train.jsonl
├── train.txt
├── valid.jsonl
└── valid.txt
Seems ok, do you still encounter the issue?
The actual issue was that libclang-6.0
is needed. Probably a dependency for some package involved.
Doing the following fixed the issue:
sudo apt-get install -y libclang-6.0-dev
Closing this issue. Thanks a lot!
Ah I see, the tokenizer supports the C++ language too. So, libclang is required. Perhaps, I should remove it since none of the downstream tasks is involved with the C++ language.
While running
prepare.sh
the followign errors are thrown for all the languages inmultilingual
directory:Any help on this?