search

openai / transformer-debugger

MIT License
4.01k stars 231 forks source link

update azure paths so that they work with no credentials #10

Closed henktillman closed 5 months ago

ianand commented 5 months ago

(thx, following, https://twitter.com/ianand/status/1767827185170300935)

hanw commented 5 months ago

Here is a unit test for the credential problem. It has something to do with blobfile.listdir

blobfile._common.Error: Encountered an error when requesting an access token: `invalid_request: AADSTS900144: The request body must contain the following parameter: 'client_id'. Trace ID: 0122581e-b632-44bc-a67d-10a375870d00 Correlation ID: 32846173-b7cc-4195-a763-1646b9860cf5 Timestamp: 2024-03-14 01:03:45Z`.  You can attempt to fix this by re-running `az login`.
import blobfile as bf

pieces_path = f"az://openaipublic/neuron-explainer/subject-models/gpt2/small/model_pieces"
piece_files = list(bf.listdir(pieces_path))
print(piece_files)

Using https://openaipublic.blob.core.windows.net makes no difference.

swj0418 commented 5 months ago

When you try to access model files in browser, such as gpt2" via https://openaipublic.blob.core.windows.net/neuron-explainer/subject-models/gpt2/small, it says that the specified blob file does not exist.

On the other hand, pre-computed activations https://openaipublic.blob.core.windows.net/neuron-explainer/gpt2_small_data/collated-activations/{layer_index}/{neuron_index}.json can be accessed via browser.

I think model paths are broken.

VincentStark commented 5 months ago

@henktillman you sir are my hero. I was about to start fixing it myself.

As a sidenote, I don't understand how this project can have 2.9k stars, and nobody took notice that it actually doesn't work for anyone outside of OAI.

hanw commented 5 months ago

Please consider turn on the issue tab. It would make it easier to discuss issues like this one.