search

Jun-jie-Huang / ExeDS

Source code and data for our paper at DaSH@EMNLP-22: "Execution-based Evaluation for Data Science Code Generation Models"
MIT License
7 stars 1 forks source link

Error while initializing BPE #3

Open IssamLaradji opened 1 year ago

IssamLaradji commented 1 year ago

Hi again,

I ran this command in Section 2.2 and I get the error in the picture below. Could you please advise?

python preprocess.py \
            --split dev \
            --file_name exeds_dev.json \
            --do_fairseq_tokenization \
            --do_gptneo \
            --token_type token \
            --context_range 3 \
            --max_code_cell_tokens 200 \
            --max_md_cell_tokens 200 \
            --max_ctx_cell_tokens 900
image
Jun-jie-Huang commented 1 year ago

The BPE tokenizer is a built-in API in transformers. I think there might be come conflicts with the module versions. You can check you environment with our requirements

pixas commented 1 year ago

Hi again,

I ran this command in Section 2.2 and I get the error in the picture below. Could you please advise?

python preprocess.py \
            --split dev \
            --file_name exeds_dev.json \
            --do_fairseq_tokenization \
            --do_gptneo \
            --token_type token \
            --context_range 3 \
            --max_code_cell_tokens 200 \
            --max_md_cell_tokens 200 \
            --max_ctx_cell_tokens 900
image

hi, do you solve this problem? I also meet the same error and I have checked the requirement