search

neuralmagic / sparsify

ML model optimization product to accelerate inference.
Apache License 2.0
320 stars 29 forks source link

Trying to apply sparsify on 1-layer transformer model #86

Closed mbetser closed 2 years ago

mbetser commented 2 years ago

I have tried the sparsify interface. When hitting run, the server crashes with the trace bellow. Any help is welcomed!

Michael

2021-11-13 00:19:27 sparsify.blueprints.jobs INFO retrieved job {'job': {'error': None, 'jobid': 'e1eb1c43eb614143b2c0a31285eca111', 'created': '2021-11-13T00:19:27.071845', 'modified': '2021-11-13T00:19:27.071870', 'type': 'CreatePerfProfileJobWorker', 'status': 'pending', 'project_id': '8c8f6df6be0d4dd18d15716bdf7ff327', 'progress': None, 'worker_args': {'model_id': 'de66e42d06af4e4786b210c0ee59b0b2', 'profile_id': 'a8005237aef943c3bf4917ce0210bd5f', 'batch_size': 1, 'core_count': 4, 'pruning_estimations': True, 'quantized_estimations': False, 'iterations_per_check': 10, 'warmup_iterations_per_check': 5}}} 10.0.0.4 - - [13/Nov/2021 00:19:27] "GET /api/jobs/e1eb1c43eb614143b2c0a31285eca111 HTTP/1.1" 200 - 2021-11-13 00:19:27 sparsify.workers.projects_profiles INFO running perf profile for project_id 8c8f6df6be0d4dd18d15716bdf7ff327 and model_id de66e42d06af4e4786b210c0ee59b0b2 and profile_id a8005237aef943c3bf4917ce0210bd5f with batch_size:1, core_count:4, pruning_estimations:True, quantized_estimations:False, iterations_per_check:10, warmup_iterations_per_check:5 DeepSparse Engine, Copyright 2021-present / Neuralmagic, Inc. version: 0.8.0 (68df72e1) (release) (optimized) (system=avx512, binary=avx512) 2021-11-13 00:19:27 sparsify.blueprints.jobs INFO getting job e1eb1c43eb614143b2c0a31285eca111 2021-11-13 00:19:27 sparsify.blueprints.jobs INFO retrieved job {'job': {'error': None, 'jobid': 'e1eb1c43eb614143b2c0a31285eca111', 'created': '2021-11-13T00:19:27.071845', 'modified': '2021-11-13T00:19:27.155796', 'type': 'CreatePerfProfileJobWorker', 'status': 'started', 'project_id': '8c8f6df6be0d4dd18d15716bdf7ff327', 'progress': {'iter_indefinite': False, 'iter_class': 'analysis', 'num_steps': 2, 'step_class': 'baseline_estimation', 'step_index': 0, 'iter_val': 0.0}, 'worker_args': {'model_id': 'de66e42d06af4e4786b210c0ee59b0b2', 'profile_id': 'a8005237aef943c3bf4917ce0210bd5f', 'batch_size': 1, 'core_count': 4, 'pruning_estimations': True, 'quantized_estimations': False, 'iterations_per_check': 10, 'warmup_iterations_per_check': 5}}} 10.0.0.4 - - [13/Nov/2021 00:19:27] "GET /api/jobs/e1eb1c43eb614143b2c0a31285eca111 HTTP/1.1" 200 - [nm_ort 7fbe5589e700 >ERROR< supported_subgraphs /home/ubuntu/build/nyann/src/onnxruntime_neuralmagic/supported/subgraphs.cc:782] ==== FAILED TO COMPILE ==== Unexpected exception message: bad optional access DeepSparse Engine, Copyright 2021-present / Neuralmagic, Inc. version: 0.8.0 (68df72e1) (release) (optimized) Date: 11-13-2021 @ 00:19:27 UTC OS: Linux linuxvm1 4.15.0-1061-azure #66-Ubuntu SMP Thu Oct 3 02:00:50 UTC 2019 Arch: x86_64 CPU: Intel(R) Xeon(R) Platinum 8272CL CPU @ 2.60GHz Vendor: GenuineIntel Cores/sockets/threads: [4, 1, 8] Available cores/sockets/threads: [4, 1, 8] L1 cache size data/instruction: 32k/32k L2 cache size: 1Mb L3 cache size: 35.75Mb Total memory: 15.6651G Free memory: 1.88776G

Assertion at /home/ubuntu/build/nyann/src/onnxruntime_neuralmagic/nm_execution_provider.cc:76

Backtrace: 0# wand::detail::abort_prefix(std::ostream&, char const, char const, int, bool, bool, unsigned long) in /home/mbetser/anaconda3/envs/optimize/lib/python3.8/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0 1# 0x00007FBE2913F285 in /home/mbetser/anaconda3/envs/optimize/lib/python3.8/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0 2# 0x00007FBE291410AE in /home/mbetser/anaconda3/envs/optimize/lib/python3.8/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0 3# 0x00007FBE2940D1C1 in /home/mbetser/anaconda3/envs/optimize/lib/python3.8/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0 4# 0x00007FBE29A5A668 in /home/mbetser/anaconda3/envs/optimize/lib/python3.8/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0 5# 0x00007FBE29A5D0A2 in /home/mbetser/anaconda3/envs/optimize/lib/python3.8/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0 6# 0x00007FBE29A603B9 in /home/mbetser/anaconda3/envs/optimize/lib/python3.8/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0 7# 0x00007FBE293EC76C in /home/mbetser/anaconda3/envs/optimize/lib/python3.8/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0 8# 0x00007FBE293F24C3 in /home/mbetser/anaconda3/envs/optimize/lib/python3.8/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0 9# 0x00007FBE293AC982 in /home/mbetser/anaconda3/envs/optimize/lib/python3.8/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0 10# 0x00007FBE293ACC05 in /home/mbetser/anaconda3/envs/optimize/lib/python3.8/site-packages/deepsparse/avx512/libonnxruntime.so.1.8.0 11# deepsparse::ort_engine::init(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, int, int, int, wand::safe_type<wand::parallel::use_current_affinity_tag, bool>, std::shared_ptr) in /home/mbetser/anaconda3/envs/optimize/lib/python3.8/site-packages/deepsparse/avx512/libdeepsparse.so 12# 0x00007FBE5FDDD7EB in /home/mbetser/anaconda3/envs/optimize/lib/python3.8/site-packages/deepsparse/avx512/deepsparse_engine.so 13# 0x00007FBE5FDDDA09 in /home/mbetser/anaconda3/envs/optimize/lib/python3.8/site-packages/deepsparse/avx512/deepsparse_engine.so 14# 0x00007FBE5FDFD986 in /home/mbetser/anaconda3/envs/optimize/lib/python3.8/site-packages/deepsparse/avx512/deepsparse_engine.so 15# 0x00007FBE5FDEAA09 in /home/mbetser/anaconda3/envs/optimize/lib/python3.8/site-packages/deepsparse/avx512/deepsparse_engine.so 16# 0x00005592DB0A07AE in /home/mbetser/anaconda3/envs/optimize/bin/python 17# _PyObject_MakeTpCall in /home/mbetser/anaconda3/envs/optimize/bin/python 18# 0x00005592DB0CAD6A in /home/mbetser/anaconda3/envs/optimize/bin/python 19# PyObject_Call in /home/mbetser/anaconda3/envs/optimize/bin/python 20# 0x00005592DB040689 in /home/mbetser/anaconda3/envs/optimize/bin/python 21# 0x00005592DB0A06C7 in /home/mbetser/anaconda3/envs/optimize/bin/python 22# 0x00007FBE42973029 in /home/mbetser/anaconda3/envs/optimize/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.cpython-38-x86_64-linux-gnu.so 23# _PyObject_MakeTpCall in /home/mbetser/anaconda3/envs/optimize/bin/python

Please email a copy of this stack trace and any additional information to: support@neuralmagic.com Aborted

jeanniefinks commented 2 years ago

Hello @mbetser Transformers and other language models are not supported at this time. I am cc-ig my colleague @bfineran as he may have a workaround to share with you. Regards, Jeannie / Neural Magic

jeanniefinks commented 2 years ago

Hi @mbetser Is there an open-source integration you could point us to or are you using a private model? The aforementioned workaround will not be viable at this time as it's not fully realized so we're happy to investigate specifically the model you had been pursuing. Thank you. Jeannie / Neural Magic

mbetser commented 2 years ago

Hi Jenni,

Thanks for your reply! Unfortunately the model is proprietary and cannot be shared easily. However a similar model could be developed from public Bert models and distilling it to one layer. Let's put it that way: if you can make it work for a bert model, there s no reason that we could not adapt the solution to our model. Thank you and happy holidays!

Michael

On Mon, Dec 27, 2021, 10:40 AM Jeannie Finks @.***> wrote:

Hi @mbetser https://github.com/mbetser Is there an open-source integration you could point us to or are you using a private model? The aforementioned workaround will not be viable at this time as it's not fully realized so we're happy to investigate specifically the model you had been pursuing. Thank you. Jeannie / Neural Magic

— Reply to this email directly, view it on GitHub https://github.com/neuralmagic/sparsify/issues/86#issuecomment-1001672481, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEN5NXQZZTTLDCJVZ3H7KXTUTCQH7ANCNFSM5H564LYA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

markurtz commented 2 years ago

Hi @mbetser, currently further work on Sparsify including the expansion of support to other models is on our medium-term roadmap. We do have Hugging Face BERT examples and recipes pushed in SparseML and available on the SparseZoo if you'd like to take a look at those and see if you can adapt them to your needs as you mention here. Here are the corresponding links:

Additionally, we have some new results for BERT and engineering improvements that will be released shortly with our 0.10 release, so stay tuned for that!

jeanniefinks commented 2 years ago

Hello @mbetser As there has been no further commentary, I am going to go ahead and close this thread out. But if you have more comments, please re-open and we'd love to chat. Lastly, if you have not starred our sparsify repo already, and you feel inclined, please do! Thank you in advance for your support! https://github.com/neuralmagic/sparsify

Best, Jeannie / Neural Magic

edwardnguyen1705 commented 2 years ago

Dear @jeanniefinks , I am on a MacOS, so I use docker container to run the app sparsify. I want to get a recipe for the nanodet-plus-m-320 (https://github.com/RangiLyu/nanodet). However, after starting to analyze the model, I received:

DeepSparse Engine, Copyright 2021-present / Neuralmagic, Inc. version: 0.10.0 (c2458ea3) (release) (optimized)
Date: 03-08-2022 @ 01:31:57 UTC
OS: Linux 97e4719f264c 5.10.76-linuxkit #1 SMP Mon Nov 8 10:21:19 UTC 2021
Arch: x86_64
CPU: Intel(R) Core(TM) i5-8279U CPU @ 2.40GHz
Vendor: GenuineIntel
Cores/sockets/threads: [4, 1, 4]
Available cores/sockets/threads: [4, 1, 4]
L1 cache size data/instruction: 32k/32k
L2 cache size: 0.25Mb
L3 cache size: 6Mb
Total memory: 2.93919G
Free memory: 0.76593G

Assertion at ./src/include/wand/utility/pyramidal/task_graph_utils.hpp:133

Backtrace:
 0# wand::detail::abort_prefix(std::ostream&, char const*, char const*, int, bool, bool, unsigned long) in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
 1# 0x00007F84525D1D08 in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
 2# 0x00007F84525D6487 in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
 3# 0x00007F84525DAB76 in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
 4# 0x00007F8452512F6F in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
 5# 0x00007F84525150A5 in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
 6# 0x00007F8452516444 in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
 7# 0x00007F8452516819 in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
 8# 0x00007F84524C7E1B in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
 9# 0x00007F8452429704 in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
10# 0x00007F8452429A32 in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
11# 0x00007F8452429B78 in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
12# 0x00007F8452429D5D in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
13# 0x00007F8452429FA8 in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
14# 0x00007F845242A010 in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
15# 0x00007F8451A2CD47 in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
16# 0x00007F8451A330CF in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
17# 0x00007F84519AB52B in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
18# 0x00007F845199B109 in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
19# 0x00007F845199C2C1 in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
20# 0x00007F84520276B8 in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
21# 0x00007F845202A0CC in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
22# 0x00007F845202D399 in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0
23# 0x00007F845197C9AB in /usr/local/lib/python3.9/site-packages/deepsparse/avx2/libonnxruntime.so.1.10.0

Please email a copy of this stack trace and any additional information to: support@neuralmagic.com
Aborted

nanodet-plus-m_320.onnx.zip

Could you take a look? Thanks for your time.

edwardnguyen1705 commented 2 years ago

Hi @mbetser , Have you solved the issue yet?