search

furiosa-ai / furiosa-models

FuriosaAI Model Zoo Project
https://furiosa-ai.github.io/furiosa-models/
Apache License 2.0
9 stars 6 forks source link

Simplify CI steps #177

Closed furiosamg closed 10 months ago

furiosamg commented 10 months ago

Built on top of #176

Changes

Profits

furiosa-infra commented 10 months ago

Pip freeze result for all

pip freeze result
aiobotocore==2.7.0
aiofiles==23.2.1
aiohttp==3.8.6
aiohttp-retry==2.8.3
aioitertools==0.11.0
aiosignal==1.3.1
amqp==5.2.0
annotated-types==0.6.0
antlr4-python3-runtime==4.9.3
appdirs==1.4.4
asttokens==2.4.1
async-timeout==4.0.3
asyncssh==2.14.1
atpublic==4.0
attrs==23.1.0
billiard==4.2.0
boto3==1.28.64
botocore==1.31.64
celery==5.3.5
certifi==2023.7.22
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.3.0
cmake==3.27.7
colorama==0.4.6
comm==0.2.0
configobj==5.0.8
contourpy==1.2.0
cryptography==41.0.5
cycler==0.12.1
Cython==3.0.5
debugpy==1.8.0
decorator==5.1.1
dictdiffer==0.9.0
diskcache==5.6.3
distro==1.8.0
dpath==2.1.6
dulwich==0.21.6
dvc==3.29.0
dvc-data==2.20.0
dvc-http==2.30.2
dvc-objects==1.2.0
dvc-render==0.6.0
dvc-s3==2.23.0
dvc-studio-client==0.15.0
dvc-task==0.3.0
entrypoints==0.4
exceptiongroup==1.1.3
executing==2.0.1
fastjsonschema==2.19.0
filelock==3.13.1
flatten-dict==0.4.2
flufl.lock==7.1.1
fonttools==4.44.1
frozenlist==1.4.0
fsspec==2023.10.0
funcy==2.0
furiosa-common==0.10.0
furiosa-models @ file:///workspace/source
furiosa-native-postprocess==0.9.0
furiosa-native-runtime==0.10.2
furiosa-quantizer==0.10.0
furiosa-quantizer-impl==0.10.1
furiosa-runtime==0.10.0
gitdb==4.0.11
GitPython==3.1.40
grandalf==0.8
gto==1.5.0
hydra-core==1.3.2
idna==3.4
importlib-metadata==6.8.0
importlib-resources==6.1.1
iniconfig==2.0.0
ipykernel==6.26.0
ipython==8.17.2
iterative-telemetry==0.0.8
jedi==0.19.1
Jinja2==3.1.2
jmespath==1.0.1
jsonschema==4.19.2
jsonschema-specifications==2023.11.1
jupyter_client==8.6.0
jupyter_core==5.5.0
kiwisolver==1.4.5
kombu==5.3.3
lit==17.0.5
markdown-it-py==3.0.0
MarkupSafe==2.1.3
matplotlib==3.8.1
matplotlib-inline==0.1.6
mdurl==0.1.2
mpmath==1.3.0
multidict==6.0.4
multipledispatch==1.0.0
nbclient==0.6.8
nbformat==5.9.2
nbmake==1.4.6
nest-asyncio==1.5.8
networkx==3.2.1
numpy==1.25.2
nvidia-cublas-cu11==11.10.3.66
nvidia-cuda-cupti-cu11==11.7.101
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==8.5.0.96
nvidia-cufft-cu11==10.9.0.58
nvidia-curand-cu11==10.2.10.91
nvidia-cusolver-cu11==11.4.0.1
nvidia-cusparse-cu11==11.7.4.91
nvidia-nccl-cu11==2.14.3
nvidia-nvtx-cu11==11.7.91
omegaconf==2.3.0
onnx==1.14.1
opencv-python-headless==4.8.0.76
orjson==3.9.10
packaging==23.2
pandas==2.0.3
parso==0.8.3
pathspec==0.11.2
pexpect==4.8.0
Pillow==10.1.0
platformdirs==3.11.0
pluggy==1.3.0
prompt-toolkit==3.0.41
protobuf==4.25.0
psutil==5.9.6
ptyprocess==0.7.0
pure-eval==0.2.2
py-cpuinfo==9.0.0
pyarrow==12.0.1
pycocotools==2.0.7
pycparser==2.21
pydantic==2.5.0
pydantic_core==2.14.1
pydot==1.4.2
pygit2==1.13.2
Pygments==2.16.1
pygtrie==2.5.0
pyparsing==3.1.1
pytest==7.4.3
pytest-asyncio==0.17.2
pytest-benchmark==4.0.0
python-dateutil==2.8.2
pytz==2023.3.post1
PyYAML==6.0.1
pyzmq==25.1.1
referencing==0.31.0
requests==2.31.0
rich==13.6.0
rpds-py==0.12.0
ruamel.yaml==0.18.5
ruamel.yaml.clib==0.2.8
s3fs==2023.10.0
s3transfer==0.7.0
scmrepo==1.4.1
semver==3.0.2
shortuuid==1.0.11
shtab==1.6.4
six==1.16.0
smmap==5.0.1
sqltrie==0.8.0
stack-data==0.6.3
sympy==1.12
tabulate==0.9.0
tomli==2.0.1
tomlkit==0.12.3
torch==2.0.1
torchvision==0.15.2
tornado==6.3.3
tqdm==4.66.1
traitlets==5.13.0
triton==2.0.0
typer==0.9.0
typing_extensions==4.8.0
tzdata==2023.3
urllib3==1.26.18
vine==5.1.0
voluptuous==0.14.0
wcwidth==0.2.10
wrapt==1.16.0
yarl==1.9.2
zc.lockfile==3.0.post1
zipp==3.17.0
furiosa-infra commented 10 months ago

Pip freeze result for all

pip freeze result
aiobotocore==2.7.0
aiofiles==23.2.1
aiohttp==3.8.6
aiohttp-retry==2.8.3
aioitertools==0.11.0
aiosignal==1.3.1
amqp==5.2.0
annotated-types==0.6.0
antlr4-python3-runtime==4.9.3
appdirs==1.4.4
asttokens==2.4.1
async-timeout==4.0.3
asyncssh==2.14.1
atpublic==4.0
attrs==23.1.0
billiard==4.2.0
boto3==1.28.64
botocore==1.31.64
celery==5.3.5
certifi==2023.7.22
cffi==1.16.0
charset-normalizer==3.3.2
click==8.1.7
click-didyoumean==0.3.0
click-plugins==1.1.1
click-repl==0.3.0
cmake==3.27.7
colorama==0.4.6
comm==0.2.0
configobj==5.0.8
contourpy==1.2.0
cryptography==41.0.5
cycler==0.12.1
Cython==3.0.5
debugpy==1.8.0
decorator==5.1.1
dictdiffer==0.9.0
diskcache==5.6.3
distro==1.8.0
dpath==2.1.6
dulwich==0.21.6
dvc==3.29.0
dvc-data==2.20.0
dvc-http==2.30.2
dvc-objects==1.2.0
dvc-render==0.6.0
dvc-s3==2.23.0
dvc-studio-client==0.15.0
dvc-task==0.3.0
entrypoints==0.4
exceptiongroup==1.1.3
executing==2.0.1
fastjsonschema==2.19.0
filelock==3.13.1
flatten-dict==0.4.2
flufl.lock==7.1.1
fonttools==4.44.1
frozenlist==1.4.0
fsspec==2023.10.0
funcy==2.0
furiosa-common==0.10.0
furiosa-models @ file:///workspace/source
furiosa-native-postprocess==0.9.0
furiosa-native-runtime==0.10.2
furiosa-quantizer==0.10.0
furiosa-quantizer-impl==0.10.1
furiosa-runtime==0.10.0
gitdb==4.0.11
GitPython==3.1.40
grandalf==0.8
gto==1.5.0
hydra-core==1.3.2
idna==3.4
importlib-metadata==6.8.0
importlib-resources==6.1.1
iniconfig==2.0.0
ipykernel==6.26.0
ipython==8.17.2
iterative-telemetry==0.0.8
jedi==0.19.1
Jinja2==3.1.2
jmespath==1.0.1
jsonschema==4.19.2
jsonschema-specifications==2023.11.1
jupyter_client==8.6.0
jupyter_core==5.5.0
kiwisolver==1.4.5
kombu==5.3.3
lit==17.0.5
markdown-it-py==3.0.0
MarkupSafe==2.1.3
matplotlib==3.8.1
matplotlib-inline==0.1.6
mdurl==0.1.2
mpmath==1.3.0
multidict==6.0.4
multipledispatch==1.0.0
nbclient==0.6.8
nbformat==5.9.2
nbmake==1.4.6
nest-asyncio==1.5.8
networkx==3.2.1
numpy==1.25.2
nvidia-cublas-cu11==11.10.3.66
nvidia-cuda-cupti-cu11==11.7.101
nvidia-cuda-nvrtc-cu11==11.7.99
nvidia-cuda-runtime-cu11==11.7.99
nvidia-cudnn-cu11==8.5.0.96
nvidia-cufft-cu11==10.9.0.58
nvidia-curand-cu11==10.2.10.91
nvidia-cusolver-cu11==11.4.0.1
nvidia-cusparse-cu11==11.7.4.91
nvidia-nccl-cu11==2.14.3
nvidia-nvtx-cu11==11.7.91
omegaconf==2.3.0
onnx==1.14.1
opencv-python-headless==4.8.0.76
orjson==3.9.10
packaging==23.2
pandas==2.0.3
parso==0.8.3
pathspec==0.11.2
pexpect==4.8.0
Pillow==10.1.0
platformdirs==3.11.0
pluggy==1.3.0
prompt-toolkit==3.0.41
protobuf==4.25.0
psutil==5.9.6
ptyprocess==0.7.0
pure-eval==0.2.2
py-cpuinfo==9.0.0
pyarrow==12.0.1
pycocotools==2.0.7
pycparser==2.21
pydantic==2.5.0
pydantic_core==2.14.1
pydot==1.4.2
pygit2==1.13.2
Pygments==2.16.1
pygtrie==2.5.0
pyparsing==3.1.1
pytest==7.4.3
pytest-asyncio==0.17.2
pytest-benchmark==4.0.0
python-dateutil==2.8.2
pytz==2023.3.post1
PyYAML==6.0.1
pyzmq==25.1.1
referencing==0.31.0
requests==2.31.0
rich==13.6.0
rpds-py==0.12.0
ruamel.yaml==0.18.5
ruamel.yaml.clib==0.2.8
s3fs==2023.10.0
s3transfer==0.7.0
scmrepo==1.4.1
semver==3.0.2
shortuuid==1.0.11
shtab==1.6.4
six==1.16.0
smmap==5.0.1
sqltrie==0.8.0
stack-data==0.6.3
sympy==1.12
tabulate==0.9.0
tomli==2.0.1
tomlkit==0.12.3
torch==2.0.1
torchvision==0.15.2
tornado==6.3.3
tqdm==4.66.1
traitlets==5.13.0
triton==2.0.0
typer==0.9.0
typing_extensions==4.8.0
tzdata==2023.3
urllib3==1.26.18
vine==5.1.0
voluptuous==0.14.0
wcwidth==0.2.10
wrapt==1.16.0
yarl==1.9.2
zc.lockfile==3.0.post1
zipp==3.17.0
furiosa-infra commented 10 months ago

da0d14b Simplify ci


-------------------------------------------------------------------------------------------------------- benchmark: 9 tests -------------------------------------------------------------------------------------------------------
Name (time in ms)                                                 Min                 Max                Mean             StdDev              Median                IQR            Outliers       OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_mlcommons_resnet50_accuracy                               1.2362 (1.0)       61.7635 (3.02)       4.8962 (1.0)       1.7076 (1.73)       4.6881 (1.0)       1.3013 (1.61)    4679;3042  204.2395 (1.0)       50000           1
test_mlcommons_ssd_mobilenet_with_native_rust_pp_accuracy      4.1251 (3.34)      32.4517 (1.59)       8.0586 (1.65)      0.9853 (1.0)        7.9969 (1.71)      0.8062 (1.0)      1085;506  124.0912 (0.61)       5000           1
test_efficientnetb0_accuracy                                   4.6108 (3.73)     446.8611 (21.86)     13.9586 (2.85)     13.9483 (14.16)     12.3959 (2.64)      3.2579 (4.04)     893;2244   71.6403 (0.35)      50000           1
test_efficientnetv2s_accuracy                                  6.9882 (5.65)     522.0111 (25.53)     17.0324 (3.48)     15.1525 (15.38)     15.3364 (3.27)      4.2780 (5.31)     883;1756   58.7117 (0.29)      50000           1
test_mlcommons_ssd_mobilenet_accuracy                          7.1682 (5.80)     233.6688 (11.43)     11.0874 (2.26)      3.3029 (3.35)      11.0824 (2.36)      0.8557 (1.06)       43;498   90.1926 (0.44)       5000           1
test_yolov5m_accuracy                                          7.4391 (6.02)      20.4434 (1.0)       10.8732 (2.22)      1.4569 (1.48)      10.6368 (2.27)      1.8300 (2.27)     2793;246   91.9693 (0.45)      10000           1
test_yolov5l_accuracy                                         10.7099 (8.66)      31.5456 (1.54)      14.7960 (3.02)      1.6855 (1.71)      14.4516 (3.08)      2.1098 (2.62)     2837;242   67.5856 (0.33)      10000           1
test_mlcommons_ssd_resnet34_with_native_rust_pp_accuracy      34.8740 (28.21)    524.0578 (25.63)     49.9398 (10.20)    12.2420 (12.42)     49.3697 (10.53)     2.1892 (2.72)       26;186   20.0241 (0.10)       5000           1
test_mlcommons_ssd_resnet34_accuracy                          92.2223 (74.60)    982.5889 (48.06)    120.6055 (24.63)    24.9294 (25.30)    116.9415 (24.94)    16.6287 (20.63)     273;171    8.2915 (0.04)       5000           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean
furiosamg commented 10 months ago

Now ready for review

hyunsik commented 10 months ago

I also prefer this way rather than building and pushing a docker image during a running CI. This PR will require an extra step to build and push the docker images outside of CI. Could you update the instruction at the wiki page?

furiosamg commented 10 months ago

I also prefer this way rather than building and pushing a docker image during a running CI. This PR will require an extra step to build and push the docker images outside of CI. Could you update the instruction at the wiki page?

Yes, someone needs to build and push the Docker image. It seems like the wiki page for this project has not been maintained for a while. Can I update the developer's guide instead?

hyunsik commented 10 months ago

Sure, please feel free to update the guide.

furiosamg commented 10 months ago

/gen-doc

github-actions[bot] commented 10 months ago

Documentation generated by PR-177 successfully! Documentation commit: https://github.com/furiosa-ai/furiosa-models/commit/3292d4e Documentation URL: https://furiosa-ai.github.io/furiosa-models/PR-177/

furiosamg commented 10 months ago

@hyunsik I also updated the Docker build & push instruction at this link