guangy10 commented 2 days ago

What does this PR do?

Adding generate support for exported model. Adding gemma2-2b to ExecuTorch with tests. Adding an integration test for gemma-2b that we've enabled already.

Additional Test in `ExecuTorch`

Running gemma2-2b E2E:

cmake-out/examples/models/llama2/llama_main --tokenizer_path=tokenizer_gemma2.bin --model_path=gemma2.pte --prompt="My name is"
I 00:00:00.001356 executorch:cpuinfo_utils.cpp:62] Reading file /sys/devices/soc0/image_version
I 00:00:00.001425 executorch:cpuinfo_utils.cpp:78] Failed to open midr file /sys/devices/soc0/image_version
I 00:00:00.001431 executorch:cpuinfo_utils.cpp:158] Number of efficient cores 4
I 00:00:00.001434 executorch:main.cpp:65] Resetting threadpool with num threads = 6
I 00:00:00.005564 executorch:runner.cpp:65] Creating LLaMa runner: model_path=gemma2.pte, tokenizer_path=tokenizer_gemma2.bin
E 00:00:03.701808 executorch:tiktoken.cpp:79] invalid tiktoken line:
I 00:00:03.701845 executorch:runner.cpp:88] Failed to load tokenizer_gemma2.bin as a Tiktoken artifact, trying BPE tokenizer
I 00:00:03.767485 executorch:runner.cpp:94] Reading metadata from model
I 00:00:03.767512 executorch:runner.cpp:119] Metadata: get_vocab_size = 256000
I 00:00:03.767515 executorch:runner.cpp:119] Metadata: get_bos_id = 2
I 00:00:03.767517 executorch:runner.cpp:117] Methond use_sdpa_with_kv_cache not found, using the default value 0
I 00:00:03.767518 executorch:runner.cpp:119] Metadata: use_sdpa_with_kv_cache = 0
I 00:00:03.767520 executorch:runner.cpp:119] Metadata: get_n_eos = 1
I 00:00:03.767521 executorch:runner.cpp:117] Methond append_eos_to_prompt not found, using the default value 0
I 00:00:03.767522 executorch:runner.cpp:119] Metadata: append_eos_to_prompt = 0
I 00:00:03.767524 executorch:runner.cpp:119] Metadata: get_max_seq_len = 123
I 00:00:03.767525 executorch:runner.cpp:117] Methond enable_dynamic_shape not found, using the default value 0
I 00:00:03.767527 executorch:runner.cpp:119] Metadata: enable_dynamic_shape = 0
I 00:00:03.767529 executorch:runner.cpp:119] Metadata: use_kv_cache = 1
I 00:00:03.767575 executorch:runner.cpp:119] Metadata: get_n_bos = 1
I 00:00:03.767604 executorch:runner.cpp:167] RSS after loading model: 0.000000 MiB (0 if unsupported)
I 00:00:04.408489 executorch:runner.cpp:234] RSS after prompt prefill: 0.000000 MiB (0 if unsupported)
My name is Lale and I love to play with my dolls. I started to play with dolls at the age of two years old. My favorite activity is dancing. I would like to help people. I would like to travel to Spain. I like to be a vet. I love to help people. I would like to travel. I would like to be. I love to travel. I would like to be. I like to travel. I would like to be. I love to travel. I would like to be. I love to travel. I would like to be. I love to travel
I 00:00:23.188418 executorch:runner.cpp:246] RSS after finishing text generation: 0.000000 MiB (0 if unsupported)
PyTorchObserver {"prompt_tokens":4,"generated_tokens":118,"model_load_start_ms":1727310065089,"model_load_end_ms":1727310068851,"inference_start_ms":1727310068851,"inference_end_ms":1727310088272,"prompt_eval_end_ms":1727310069492,"first_token_ms":1727310069492,"aggregate_sampling_time_ms":180,"SCALING_FACTOR_UNITS_PER_SECOND":1000}
I 00:00:23.188436 executorch:stats.h:84]    Prompt Tokens: 4    Generated Tokens: 118
I 00:00:23.188438 executorch:stats.h:90]    Model Load Time:        3.762000 (seconds)
I 00:00:23.188440 executorch:stats.h:100]   Total inference time:       19.421000 (seconds)      Rate:  6.075897 (tokens/second)
I 00:00:23.188442 executorch:stats.h:108]       Prompt evaluation:  0.641000 (seconds)       Rate:  6.240250 (tokens/second)
I 00:00:23.188444 executorch:stats.h:119]       Generated 118 tokens:   18.780000 (seconds)      Rate:  6.283280 (tokens/second)
I 00:00:23.188446 executorch:stats.h:127]   Time to first generated token:  0.641000 (seconds)
I 00:00:23.188480 executorch:stats.h:134]   Sampling time over 122 tokens:  0.180000 (seconds)

Before submitting

[x] Did you read the contributor guideline, Pull Request section?
[x] Was this discussed/approved via a Github issue or the forum? Please add a link to it if that's the case. #33709
[x] Did you make sure to update the documentation with your changes? Here are the documentation guidelines, and here are tips on formatting docstrings.
[x] Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

@ArthurZucker @gante @amyeroberts @qubvel

guangy10 commented 1 day ago

Why am I start getting fetch_tests - Unauthorized error on all the PRs I'm creating?

LysandreJik commented 1 day ago

This seems to be happening more often, @ydshieh would you know what might be happening? Otherwise I'll reach out to the CircleCI team, this is hindering work on the repo

LysandreJik commented 1 day ago

I've asked the CircleCI team @guangy10, very sorry for the inconvenience.

ydshieh commented 1 day ago

Might be related to

Allow CI could be run on private forked repositories (e.g. new model additions) (#33594)

But not happening on all external contributor's PRs. Strange

ydshieh commented 1 day ago

Could you first try ..?

https://support.circleci.com/hc/en-us/articles/360048210711-How-to-Refresh-User-Permissions

ydshieh commented 1 day ago

Another thing to check

If you're following the fork instead of the upstream repo

A user who submits a pull request to your repository from a fork, but no pipeline is triggered with the pull request. This can happen when the user is following the project fork on their personal account rather than the project itself on CircleCI.

This will cause the jobs to trigger under the user's personal account. If the user is following a fork of the repository on CircleCI, we will only build on that fork and not the parent, so the parent’s PR will not get status updates. 

In these cases, the user unfollows their fork of the project on CircleCI. This will trigger their jobs to run under the organization when they submit pull requests. Those users can optionally follow the source project if they wish to see the pipelines.

HuggingFaceDocBuilderDev commented 1 day ago

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

guangy10 commented 1 day ago

Could you first try ..?

https://support.circleci.com/hc/en-us/articles/360048210711-How-to-Refresh-User-Permissions

Weird. Used to work fine on my old PRs. Tried "Refresh Permission". Let's see if it can be unblocked.

@ydshieh It's getting worse after re-fresh the permissions. check_circleci_user starts failing.

guangy10 commented 1 day ago

Another thing to check

If you're following the fork instead of the upstream repo

A user who submits a pull request to your repository from a fork, but no pipeline is triggered with the pull request. This can happen when the user is following the project fork on their personal account rather than the project itself on CircleCI.

This will cause the jobs to trigger under the user's personal account. If the user is following a fork of the repository on CircleCI, we will only build on that fork and not the parent, so the parent’s PR will not get status updates. 

In these cases, the user unfollows their fork of the project on CircleCI. This will trigger their jobs to run under the organization when they submit pull requests. Those users can optionally follow the source project if they wish to see the pipelines.

"In these cases, the user unfollows their fork of the project on CircleCI. " I have no idea how to unfollow my fork on CircleCI. I don't even see if I'm following transformers on CircleCI. I created a CircleCI account with exact email and linked to my github, and I can't see any project I'm following..

guangy10 commented 1 day ago

https://support.circleci.com/hc/en-us/articles/360008097173-Troubleshooting-why-pull-requests-are-not-triggering-jobs-on-my-organization

so please be sure "Build forked pull requests" is enabled in Project Settings > Advanced

@ydshieh Their Wiki is so bad. I can't find this setting from anywhere.

ydshieh commented 14 hours ago

Hmm, weird CircleCI issue ...

Could you check a last time:

top-left (like in the image): organization : switch to your own org, follow transformers there. Optionally: Then switch to huggingface (if you can find it) and follow transformers there

If still not working, I could try to push a commit.

huggingface / transformers

Generate using exported model and enable gemma2-2b in ExecuTorch #33707

What does this PR do?

Additional Test in `ExecuTorch`

Before submitting

Who can review?

huggingface / transformers

Generate using exported model and enable gemma2-2b in ExecuTorch #33707

What does this PR do?

Additional Test in ExecuTorch

Before submitting

Who can review?

Additional Test in `ExecuTorch`