search

vitrivr / cineast

Cineast is a multi-feature content-based mulitmedia retrieval engine. It is capable of retrieving images, audio- and video sequences as well as 3d models based on edge or color sketches, textual descriptions and example objects.
MIT License
57 stars 50 forks source link

Model-Loading in VisualTextCoEmbedding is not Thread-Safe #311

Closed silvanheller closed 2 years ago

silvanheller commented 2 years ago

Similar to the discussion in #303, the model loading for VisualTextCoEmbedding is also not thread-safe - consider the following output when parallel queries are executed on a freshly launched Cineast instance. The design considerations are the same as in #303.

2022-05-17 13:51:43.585 [pool-4-thread-VisualTextCoEmbedding] DEBUG o.v.c.s.r.RetrievalTask - starting VisualTextCoEmbedding                                                      [67/1487]2022-05-17 13:51:46.571402: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: resources/VisualTextCoEmbedding/universal-sentence-encoder_4
2022-05-17 13:51:46.571450: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: resources/VisualTextCoEmbedding/universal-sentence-encoder_4
2022-05-17 13:51:46.571483: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: resources/VisualTextCoEmbedding/universal-sentence-encoder_4
2022-05-17 13:51:46.571542: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: resources/VisualTextCoEmbedding/universal-sentence-encoder_4
2022-05-17 13:51:46.571569: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: resources/VisualTextCoEmbedding/universal-sentence-encoder_4
2022-05-17 13:51:46.774179: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:107] Reading meta graph with tags { serve }
2022-05-17 13:51:46.774257: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:148] Reading SavedModel debug info (if present) from: resources/VisualTextCoEmbedding/universal-sentence-encoder_4
2022-05-17 13:51:46.774733: I external/org_tensorflow/tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-05-17 13:51:46.779861: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:107] Reading meta graph with tags { serve }
2022-05-17 13:51:46.779889: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:148] Reading SavedModel debug info (if present) from: resources/VisualTextCoEmbedding/universal-sentence-encoder_4
2022-05-17 13:51:46.794457: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:107] Reading meta graph with tags { serve }
2022-05-17 13:51:46.794520: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:148] Reading SavedModel debug info (if present) from: resources/VisualTextCoEmbedding/universal-sentence-encoder_4
2022-05-17 13:51:46.795408: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:107] Reading meta graph with tags { serve }
2022-05-17 13:51:46.795475: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:148] Reading SavedModel debug info (if present) from: resources/VisualTextCoEmbedding/universal-sentence-encoder_4
2022-05-17 13:51:46.795603: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:107] Reading meta graph with tags { serve }
2022-05-17 13:51:46.795648: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:148] Reading SavedModel debug info (if present) from: resources/VisualTextCoEmbedding/universal-sentence-encoder_4
2022-05-17 13:51:47.140321: W external/org_tensorflow/tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory
2022-05-17 13:51:47.140368: W external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-05-17 13:51:47.323465: W external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-05-17 13:51:47.324322: W external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-05-17 13:51:47.326991: W external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-05-17 13:51:47.327872: W external/org_tensorflow/tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-05-17 13:51:47.918466: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:228] Restoring SavedModel bundle.
2022-05-17 13:51:47.930779: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:228] Restoring SavedModel bundle.
2022-05-17 13:51:47.930922: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:228] Restoring SavedModel bundle.
2022-05-17 13:51:47.930989: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:228] Restoring SavedModel bundle.
2022-05-17 13:51:47.931125: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:228] Restoring SavedModel bundle.
2022-05-17 13:51:49.912884: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:212] Running initialization op on SavedModel bundle at path: resources/VisualTextCoEmbedding/universal-sentence-encoder_4
2022-05-17 13:51:50.135819: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:212] Running initialization op on SavedModel bundle at path: resources/VisualTextCoEmbedding/universal-sentence-encoder_4
2022-05-17 13:51:50.150851: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:212] Running initialization op on SavedModel bundle at path: resources/VisualTextCoEmbedding/universal-sentence-encoder_4
2022-05-17 13:51:50.152102: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:212] Running initialization op on SavedModel bundle at path: resources/VisualTextCoEmbedding/universal-sentence-encoder_4
2022-05-17 13:51:50.152686: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:212] Running initialization op on SavedModel bundle at path: resources/VisualTextCoEmbedding/universal-sentence-encoder_4
2022-05-17 13:51:53.351812: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:301] SavedModel load for tags { serve }; Status: success: OK. Took 6780407 microseconds.
2022-05-17 13:51:53.385795: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:301] SavedModel load for tags { serve }; Status: success: OK. Took 6814362 microseconds.
2022-05-17 13:51:53.478506: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:301] SavedModel load for tags { serve }; Status: success: OK. Took 6907114 microseconds.
2022-05-17 13:51:53.651861: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:301] SavedModel load for tags { serve }; Status: success: OK. Took 7080439 microseconds.
2022-05-17 13:51:53.741032: I external/org_tensorflow/tensorflow/cc/saved_model/loader.cc:301] SavedModel load for tags { serve }; Status: success: OK. Took 7169610 microseconds.
2022-05-17 13:51:54.335898: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: resources/VisualTextCoEmbedding/text-co-embedding
2022-05-17 13:51:54.335984: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: resources/VisualTextCoEmbedding/text-co-embedding
2022-05-17 13:51:54.336106: I external/org_tensorflow/tensorflow/cc/saved_model/reader.cc:43] Reading SavedModel from: resources/VisualTextCoEmbedding/text-co-embedding
Spiess commented 2 years ago

Thanks for catching this, I have fixed this and will push the respective branch when ready.