issues
search
google
/
JetStream
JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).
Apache License 2.0
193
stars
24
forks
source link
issues
Newest
Newest
Most commented
Recently updated
Oldest
Least commented
Least recently updated
Understanding the intuition behind `request-rate`
#137
hosseinsarshar
opened
3 days ago
0
Change the default message for requester.py and remove mlperf 4.1 install for proxy version support.
#136
zhihaoshan-google
closed
1 week ago
0
Support completions API
#135
nstogner
opened
1 week ago
0
remove excessive logs in production run by changing from DEBUG to INFO
#134
jwyang-google
closed
2 weeks ago
0
Add an optional parameter for sampling in prefill / sample.
#133
qihqi
closed
2 weeks ago
1
Update JetStream instructions
#132
yeandy
closed
3 weeks ago
0
Clean up Model Conversion Script
#131
yeandy
opened
4 weeks ago
2
Update deps file
#130
JoeZijunZhou
closed
1 month ago
0
Standalone JetStream removes pinned deps
#129
JoeZijunZhou
closed
1 month ago
0
Added `jetstream_total_tokens_in_current_batch` metric
#128
Bslabe123
opened
1 month ago
0
Refactor Prometheus Metrics Logic and Added to Docs
#127
Bslabe123
opened
1 month ago
0
Manual model warmup to resolve AOT model warmup performance degradation
#126
vivianrwu
closed
1 month ago
3
Makefile
#125
Bslabe123
closed
1 month ago
1
Add `jetstream_request_success_count` metric
#124
Bslabe123
closed
1 month ago
1
Request input/output size metrics
#123
Bslabe123
closed
1 month ago
0
Performance optimized interleaved mode JetStream server
#122
JoeZijunZhou
opened
1 month ago
2
Various request time metrics
#121
Bslabe123
closed
1 month ago
1
when to support gpu?
#120
Mddct
opened
1 month ago
1
Free engine resource for the slot after finished one request decoding
#119
FanhaiLu1
closed
1 month ago
0
Add `jetstream_server_startup_latency` metric
#118
Bslabe123
closed
1 month ago
0
Fix benchmark script for saving benchmark result
#117
lsy323
closed
1 month ago
0
del prefill_result & update dev image
#116
morgandu
closed
1 month ago
0
Add http server to JetStream
#115
JoeZijunZhou
closed
1 month ago
0
image fix
#114
morgandu
closed
2 months ago
0
Update images for mlperf
#113
morgandu
closed
2 months ago
0
Cleanup orchestrator proto
#112
JoeZijunZhou
closed
2 months ago
0
Bump zipp from 3.17.0 to 3.19.1 in the pip group
#111
dependabot[bot]
closed
2 months ago
0
Bump certifi from 2024.2.2 to 2024.7.4 in the pip group
#110
dependabot[bot]
closed
2 months ago
0
Add loadgen in dev image
#109
morgandu
closed
2 months ago
0
change the detokenization thread to return the actual eos token.
#108
jwyang-google
closed
2 months ago
0
Update docs with metrics observation instructions
#107
Bslabe123
closed
2 months ago
0
Update docs for benchmark warmup mode
#106
JoeZijunZhou
closed
2 months ago
0
Prefill return first token
#105
jwyang-google
closed
2 months ago
0
Bump urllib3 from 2.2.0 to 2.2.2 in the pip group across 1 directory
#104
dependabot[bot]
closed
2 months ago
0
Added `jetstream_transfer_backlog_size` and `jetstream_generate_backlog_size` metrics
#103
Bslabe123
closed
2 months ago
0
Change `jetstream_slots_available_percentage` to `jetstream_slots_used_percentage`
#102
Bslabe123
closed
3 months ago
0
Add profiling server for proxy backend
#101
zhihaoshan-google
closed
3 months ago
0
Add inference sampling utils in JetStream
#100
JoeZijunZhou
closed
3 months ago
0
Add ssh port forward support for profile readme
#99
FanhaiLu1
closed
3 months ago
0
Minor fix
#98
morgandu
closed
3 months ago
0
Add tensorboard plugin dep for remote access
#97
JoeZijunZhou
closed
3 months ago
0
Update benchmark config for xlml automation
#96
morgandu
closed
3 months ago
0
Release v0.2.2
#95
JoeZijunZhou
closed
3 months ago
0
Enable JetStream Standalone Server
#94
JoeZijunZhou
opened
3 months ago
1
chore: update model_ckpt_conversion.sh
#93
eltociear
closed
3 months ago
2
Model warmup support with AOT and endpoint for JetStream
#92
vivianrwu
closed
2 months ago
2
Ensure server warmup before benchmark
#91
JoeZijunZhou
closed
3 months ago
4
Add healthcheck support for JetStream
#90
vivianrwu
closed
3 months ago
1
Add JetStream E2E test CI
#89
JoeZijunZhou
closed
3 months ago
0
Remove jax dependencies in JetStream
#88
FanhaiLu1
opened
3 months ago
0
Next