Open martenrichter opened 3 months ago
Hi @martenrichter,
Could you please confirm whether the issue has been resolved on your end, or if you still require assistance from our end?
Thank you!!
Well, yes and no.
I managed to patch the library and solve the issue by using a custom build of the package.
The patch is the PR listed in this issue.
Your colleague said he will integrate the changes manually, which makes sense as most changes come from upstream.
I just tried an install (npm install @media-pipe/tasks-vision
), and the version seems to be unchanged, not including the necessary changes. However, I have not checked all recent commits.
So, the problem persists, and I think a fix would benefit other users. (Of course, I will be happy to help.)
Hi @martenrichter,
Thank you for confirming. I will bring this to the attention of our team. Please allow us some time, and we will update you through the same thread once we have any further information available.
Hi @kuaashish, Sure, no problem; I have no hurry, as it works for me...
any update on this , trying to use tasks-vision on web worker
We can try to apply your patch but our build pipeline is pretty complicated due to how our internal JS compiler is set up. While the external build should be fully functional, we are using a Google-internal compiler to further reduce our binary size.
The changes to the compilation were just necessary so that import()
is not replaced with sth else ( if I remember correctly), so the question is if one can say to the compiler to leave it untouched?
Hi, any progress on this? Any workarounds? Thanks!
You can use the fork in my PR and compile it yourself. Or you can use the precompiled package from the fork, which I use as a workaround (but I have only done it for the vision package): https://github.com/fails-components/lectureapp/blob/master/packages/vision_pkg.tgz Of course, you should refer to the source code that led to the compilation. I have no intention of updating it, as I assume it would not take that long for it to be fixed upstream.
I am using the package from @martenrichter, however there is an issue with the face mesh's when running from a worker where the worker doesn't understand: CanvasRenderingContext2D
in the DrawingUtils contructor. See here: https://github.com/google-ai-edge/mediapipe/blob/c37721b9df27b03da34ff304696c2ed3c6d2b4cf/mediapipe/tasks/web/vision/core/drawing_utils.ts#L151
Understandably a worker would know about main thread rendering types like CanvasRenderingContext2D (as they instead use OffscreenCanvas equivalents).
My workaround is:
export class DrawingUtilsPatched extends DrawingUtils{
constructor(args: any){
self.CanvasRenderingContext2D = DrawingUtils;
super(args);
}
}
That is not surprising, as I use only the segmenter. I would guess upstream needs to change
if (cpuOrGpuGontext instanceof CanvasRenderingContext2D ||
cpuOrGpuGontext instanceof OffscreenCanvasRenderingContext2D) {
to
if ((globalThis.CanvasRenderingContext2D && cpuOrGpuGontext instanceof CanvasRenderingContext2D) ||
(globalThis.OffscreenCanvasRenderingContext2D && cpuOrGpuGontext instanceof OffscreenCanvasRenderingContext2D)) {
for being saved. You can, of course, change it in the patched branch and build your own package inside the container; it is not that difficult. (Do not ask me how; I have already forgotten the building commands.) But I am glad, that I am not the only one with this scenario of using it inside a worker.
Any updates on this? It's been with no fix for a while. Any official patching will be much appreciated
Have I written custom code (as opposed to using a stock example script provided in MediaPipe)
None
OS Platform and Distribution
Web
MediaPipe Tasks SDK version
master branch
Task name (e.g. Image classification, Gesture recognition etc.)
Image classification but should apply to all tasks
Programming Language and version (e.g. C++, Python, Java)
Javascript
Describe the actual behavior
When loading the code I get an importScript exception
Describe the expected behaviour
That it also works from workers loaded as module
Standalone code/steps you may have used to try to get what you need
If your worker is loaded as a worker, e.g.:
You will get an error saying that importScripts is not supported. Note, as framework with bundlers such as vite, convert you commonJS module to ESM module, converting the worker to commonJS is not an option.
However, I had terrible luck, as the typescript compiler in the project transpiles everything to commonJS and replaces the
import
withresolve,
which is a dead end. (At least after a few hours, I did not find a suitable workaround.) What is the reason for compiling with TypeScript to CommonJS and converting it via rollup to an ES module, anyway? So, I am here asking for help or guidance on proceeding. Changing the overall compiling workflow and supplying a patch without asking would be a bad style.