openai / gpt-2-output-dataset

Dataset of GPT-2 outputs for research in detection, biases, and more
MIT License
1.93k stars 548 forks source link

Is this capable of running on Mac M1 architecture? #36

Open nanocrowd opened 1 year ago

nanocrowd commented 1 year ago

I'm using a Mac M1 mini and am unable to install the requirements.

Successful at getting everything downloaded, changed the requirements.txt file:

transformers==2.9.1

Then installed the requirements with:

pip install -r requirements.txt

The install works up until:

error[E0463]: can't find crate forcore | = note: thex86_64-apple-darwintarget may not be installed = help: consider downloading the target withrustup target add x86_64-apple-darwin = help: consider building the standard library from source withcargo build -Zbuild-std`

  error[E0463]: can't find crate for compiler_builtins`

downloaded the target with:

rustup target add x86_64-apple-darwin

Repeated install of requirements.txt:

pip install -r requirements.txt

Same error as above - tried to build the standard library from source:

cargo build -Zbuild-std

An error was generated:

error: the-Zflag is only accepted on the nightly channel of Cargo, but this is thestablechannel

divdasani commented 1 year ago

+1 experiencing the same issue

hugoroussel commented 1 year ago

+1 Any updates on this?

nanocrowd commented 1 year ago

@divdasani @hugoroussel - yes. Updates.

I don't really know where to start. I'll open with I AM A HUGE NOOB.

I started the process by testing on an Intel laptop with Windows 10. It also failed, so I don't think the problems are isolated to Mac M1 machines.

To resolve this on the Windows 10 laptop, I did two things:

  1. I downgraded my python version from v3.11 to v.3.8.10
  2. I added an entry in the requirements.txt file tokenizers==0.7.0
  3. I changed the version of transformers to 2.9.1 transformers==2.9.1

Once those changes were applied to the Windows 10 laptop, the install and server startup worked:

  1. pip install -r requirements.txt
  2. python -m detector.server detector-base.pt

This is the full requirements.txt file:

transformers==2.9.1 fire>=0.2.1 requests>=2.22.0 tqdm>=4.32.2 torch>=1.2.0 tensorboard>=1.14.0 tokenizers==0.7.0

The situation with the Mac M1 was not as clear cut. I made all the changes as explained above for the Windows 10 laptop. However the install failed big time.

First, a rust compiler had to be installed. I'm not a rust developer, so I followed these instructions:

1) Open a terminal window. 2) Download and run the Rust installer.

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

The problems continued, but at least they were different problems.

The Rust compiler threw errors... regarding itself.

TO BE CLEAR, I am not a Rust developer. This is the first time I've done anything with Rust.

The pip install started with these error messages:

error[E0557]: feature has been removed --> /Users/myusername/.cargo/registry/src/github.com-1ecc6299db9ec823/lock_api-0.3.4/src/lib.rs:91:42

To get around this, I commented out the code in this file at that line.

Then the pip install threw this error:

error[E0557]: feature has been removed --> /Users/myusername/.cargo/registry/src/github.com-1ecc6299db9ec823/parking_lot-0.10.2/src/lib.rs:14:42

I commented out the code in this file at that line.

At this point, the pip install completed, and I am now able to run the output detecter server.

What I had to do was dodgy, to say the least. YMMV.

Edit:

Note that after installing the python v3.8.10, the command line versions of the python and pip commands changed from:

1) python changed to python3 2) pip changed to pip3