Open colljos opened 4 weeks ago
Now got this working by tweaking the Python install command in the python-container
sub-directory Dockerfile
as follows:
RUN python3 -m venv tmp; export PATH=$PATH:tmp/bin/activate:/Users/colljos/Library/Python/3.9/lib/python/site-packages; python3 -m pip install sentence-transformers jupyter pytest
Output of running docker compose run download-model
:
Joses-MacBook-Pro:hazelcast-image-search colljos$ docker compose run download-model
[+] Building 1657.2s (10/10) FINISHED docker:desktop-linux
=> [download-model internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 321B 0.0s
=> [download-model internal] load metadata for docker.io/library/python:3 1.7s
=> [download-model auth] library/python:pull token for registry-1.docker.io 0.0s
=> [download-model internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [download-model 1/4] FROM docker.io/library/python:3@sha256:bc78d3c007f86dbb87d711b8b082d9d564b8025487e780d24ccb8581d83ef8b0 172.8s
=> => resolve docker.io/library/python:3@sha256:bc78d3c007f86dbb87d711b8b082d9d564b8025487e780d24ccb8581d83ef8b0 0.0s
=> => sha256:b2b31b28ee3c96e96195c754f8679f690db4b18e475682d716122016ef056f39 49.58MB / 49.58MB 56.6s
=> => sha256:47db274575b70d4943bb3d408cdd590b8285b5811f10c2e7167ba7f0cada9f2c 2.32kB / 2.32kB 0.0s
=> => sha256:c41ea82733659810377a74cad7ee2a4d1d6f582395de8d12091be78c315cfee2 6.05kB / 6.05kB 0.0s
=> => sha256:2112e5e7c3ff699043b282f1ff24d3ef185c080c28846f1d7acc5ccf650bc13d 64.39MB / 64.39MB 92.8s
=> => sha256:bc78d3c007f86dbb87d711b8b082d9d564b8025487e780d24ccb8581d83ef8b0 9.72kB / 9.72kB 0.0s
=> => sha256:c3cc7b6f04730c072f8b292917e0d95bb886096a2b2b1781196170965161cd27 24.06MB / 24.06MB 27.0s
=> => sha256:af247aac076473044d24960a352a8ec6f154cf0a28f4fbf35fe5d43b52687ba2 211.29MB / 211.29MB 160.4s
=> => sha256:ef45f15f570bfc171a0a3444eb77562e1fdf865d7339942ce60eafbc8c84f198 6.16MB / 6.16MB 63.9s
=> => extracting sha256:b2b31b28ee3c96e96195c754f8679f690db4b18e475682d716122016ef056f39 3.4s
=> => extracting sha256:c3cc7b6f04730c072f8b292917e0d95bb886096a2b2b1781196170965161cd27 0.9s
=> => sha256:4d87d670c3ff8f79d31e0987e6da501ab7f7b5902398c8b3bfbd6c4566a54f26 27.19MB / 27.19MB 95.1s
=> => extracting sha256:2112e5e7c3ff699043b282f1ff24d3ef185c080c28846f1d7acc5ccf650bc13d 4.0s
=> => sha256:3ed3f00b0d2c8d764889e205f62819c43c3548445d2b31da54527d5b68248978 248B / 248B 93.3s
=> => extracting sha256:af247aac076473044d24960a352a8ec6f154cf0a28f4fbf35fe5d43b52687ba2 10.3s
=> => extracting sha256:ef45f15f570bfc171a0a3444eb77562e1fdf865d7339942ce60eafbc8c84f198 0.4s
=> => extracting sha256:4d87d670c3ff8f79d31e0987e6da501ab7f7b5902398c8b3bfbd6c4566a54f26 1.3s
=> => extracting sha256:3ed3f00b0d2c8d764889e205f62819c43c3548445d2b31da54527d5b68248978 0.0s
=> [download-model 2/4] RUN python3 -m venv tmp; export PATH=/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:tmp/bin/activate:/Users 1426.1s
=> [download-model 3/4] WORKDIR /root 0.1s
=> [download-model 4/4] RUN git clone https://github.com/hazelcast/hazelcast-python-client.git 3.9s
=> [download-model] exporting to image 52.3s
=> => exporting layers 52.3s
=> => writing image sha256:8c2615b4a4fcd43cebaf8e741cdca684a12d2a0cfbd8f919cebf6a7b5a80fbbc 0.0s
=> => naming to docker.io/library/hazelcast-image-search-download-model 0.0s
=> [download-model] resolving provenance for metadata file 0.0s
downloading model ...
modules.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 122/122 [00:00<00:00, 470kB/s]
config_sentence_transformers.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████| 116/116 [00:00<00:00, 466kB/s]
README.md: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.91k/1.91k [00:00<00:00, 12.7MB/s]
0_CLIPModel/config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 4.03k/4.03k [00:00<00:00, 14.0MB/s]
0_CLIPModel/preprocessor_config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████| 316/316 [00:00<00:00, 2.10MB/s]
0_CLIPModel/special_tokens_map.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████| 389/389 [00:00<00:00, 4.00MB/s]
0_CLIPModel/tokenizer_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████| 604/604 [00:00<00:00, 5.63MB/s]
0_CLIPModel/merges.txt: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 525k/525k [00:00<00:00, 932kB/s]
0_CLIPModel/vocab.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 961k/961k [00:01<00:00, 783kB/s]
pytorch_model.bin: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 605M/605M [04:44<00:00, 2.13MB/s]
done. saving ...n: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 605M/605M [04:44<00:00, 2.40MB/s]
done
Joses-MacBook-Pro:hazelcast-image-search colljos$ pip --version
pip 24.2 from /Users/colljos/Library/Python/3.9/lib/python/site-packages/pip (python 3.9)
Joses-MacBook-Pro:hazelcast-image-search colljos$ cargo --version
cargo 1.82.0
I am now experiencing a run-time exception upon executing the Deploy Pipeline step (4):
deploy the pipeline: docker compose run submit-image-loader
Output:
Joses-MacBook-Pro:hazelcast-image-search colljos$ docker compose run submit-image-loader
WARN[0000] Found orphan containers ([hazelcast-image-search-download-model-run-9f897877ad42]) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Verbose mode is on, setting logging level to INFO
Submitting JAR '/project/image-ingest-pipeline/target/image-ingest-pipeline-1.0-SNAPSHOT.jar' with arguments [/project/www, .jpg, http://www:8000, /project/image-embedding-service, clip_image_encoder]
17:41:04.811 [ INFO] [c.h.i.c.AbstractConfigLocator] Loading configuration '/opt/hazelcast/config/hazelcast-client.xml' from System property 'hazelcast.client.config'
17:41:04.840 [ INFO] [c.h.i.c.AbstractConfigLocator] Using configuration file at /opt/hazelcast/config/hazelcast-client.xml
17:41:05.767 [ INFO] [c.h.c.i.s.ClientInvocationService] hz.client_1 [dev] [5.5.2] Running with 2 response threads, dynamic=true
17:41:05.837 [ INFO] [c.h.c.LifecycleService] hz.client_1 [dev] [5.5.2] HazelcastClient 5.5.2 (20241014 - 66a3eb8) is STARTING
17:41:05.841 [ INFO] [c.h.c.LifecycleService] hz.client_1 [dev] [5.5.2] HazelcastClient 5.5.2 (20241014 - 66a3eb8) is STARTED
17:41:05.907 [ INFO] [c.h.c.i.c.ClientConnectionManager] hz.client_1 [dev] [5.5.2] Trying to connect to cluster: dev
17:41:05.914 [ INFO] [c.h.c.i.c.ClientConnectionManager] hz.client_1 [dev] [5.5.2] Trying to connect to [hz]:5701
17:41:05.945 [ INFO] [c.h.c.i.s.ClientClusterService] hz.client_1 [dev] [5.5.2]
Members [1] {
Member [172.24.0.2]:5701 - ba03f9d3-cda1-4651-be4f-1a82cf8f34ae
}
17:41:05.947 [ INFO] [c.h.c.LifecycleService] hz.client_1 [dev] [5.5.2] HazelcastClient 5.5.2 (20241014 - 66a3eb8) is CLIENT_CONNECTED
17:41:05.948 [ INFO] [c.h.c.i.c.ClientConnectionManager] hz.client_1 [dev] [5.5.2] Authenticated with server [172.24.0.2]:5701:ba03f9d3-cda1-4651-be4f-1a82cf8f34ae, server version: 5.5.2, local address: /172.24.0.6:47699
17:41:05.951 [ INFO] [c.h.i.d.Diagnostics] hz.client_1 [dev] [5.5.2] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
17:41:06.081 [ INFO] [c.h.c.i.s.ClientStatisticsService] Client statistics is enabled with period 5 seconds.
17:41:06.110 [ INFO] [c.h.i.i.e.CommandLineExecuteJar] Found mainClassName :"hazelcast.platform.labs.image.similarity.ImageIngestPipeline" and main method
17:41:06.613 [ERROR] [c.h.i.i.e.CommandLineExecuteJar] Exception caught while executing the jar
java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:118)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.hazelcast.instance.impl.executejar.CommandLineExecuteJar.invokeMain(CommandLineExecuteJar.java:117)
at com.hazelcast.instance.impl.executejar.CommandLineExecuteJar.executeJar(CommandLineExecuteJar.java:83)
at com.hazelcast.instance.impl.HazelcastBootstrap.executeJarOnCLI(HazelcastBootstrap.java:96)
at com.hazelcast.client.console.HazelcastCommandLine.submit(HazelcastCommandLine.java:196)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at picocli.CommandLine.executeUserObject(CommandLine.java:2070)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at picocli.CommandLine$RunAll.recursivelyExecuteUserObject(CommandLine.java:2540)
at picocli.CommandLine$RunAll.recursivelyExecuteUserObject(CommandLine.java:2542)
at picocli.CommandLine$RunAll.handle(CommandLine.java:2536)
at picocli.CommandLine$RunAll.handle(CommandLine.java:2496)
at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:2268)
at picocli.CommandLine.parseWithHandlers(CommandLine.java:2668)
at com.hazelcast.client.console.HazelcastCommandLine.runCommandLine(HazelcastCommandLine.java:557)
at com.hazelcast.client.console.HazelcastCommandLine.main(HazelcastCommandLine.java:134)
Caused by: java.lang.IllegalArgumentException: These transforms have nothing attached to them: [Watch for Changes]
at com.hazelcast.jet.impl.pipeline.Planner.validateNoLeakage(Planner.java:244)
at com.hazelcast.jet.impl.pipeline.Planner.createDag(Planner.java:85)
at com.hazelcast.jet.impl.pipeline.PipelineImpl.toDag(PipelineImpl.java:133)
at com.hazelcast.jet.impl.JobCoordinationService.lambda$submitJob$1(JobCoordinationService.java:259)
at com.hazelcast.jet.impl.JobCoordinationService.lambda$submitToCoordinatorThread$60(JobCoordinationService.java:1501)
at com.hazelcast.jet.impl.JobCoordinationService.lambda$submitToCoordinatorThread$61(JobCoordinationService.java:1522)
at com.hazelcast.internal.util.executor.CompletableFutureTask.run(CompletableFutureTask.java:64)
at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.lang.Thread.run(Thread.java:1583)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
at com.hazelcast.internal.util.executor.PoolExecutorThreadFactory$ManagedThread.executeRun(PoolExecutorThreadFactory.java:89)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111)
17:41:06.615 [ INFO] [c.h.c.LifecycleService] hz.client_1 [dev] [5.5.2] HazelcastClient 5.5.2 (20241014 - 66a3eb8) is SHUTTING_DOWN
17:41:06.643 [ INFO] [c.h.c.i.c.ClientConnectionManager] hz.client_1 [dev] [5.5.2] Removed connection to endpoint: [172.24.0.2]:5701:ba03f9d3-cda1-4651-be4f-1a82cf8f34ae, connection: ClientConnection{alive=false, connectionId=1, channel=NioChannel{/172.24.0.6:47699->hz/172.24.0.2:5701}, remoteAddress=[172.24.0.2]:5701, remoteUuid=ba03f9d3-cda1-4651-be4f-1a82cf8f34ae, clusterUuid=78cdcc19-ec89-4706-89de-af20f78f1762, lastReadTime=2024-11-18 17:41:06.609, lastWriteTime=2024-11-18 17:41:06.542, closedTime=2024-11-18 17:41:06.617}
17:41:06.644 [ INFO] [c.h.c.LifecycleService] hz.client_1 [dev] [5.5.2] HazelcastClient 5.5.2 (20241014 - 66a3eb8) is CLIENT_DISCONNECTED
17:41:06.648 [ INFO] [c.h.c.LifecycleService] hz.client_1 [dev] [5.5.2] HazelcastClient 5.5.2 (20241014 - 66a3eb8) is SHUTDOWN
picocli.CommandLine$ExecutionException: Error while calling command (public void com.hazelcast.client.console.HazelcastCommandLine.submit(com.hazelcast.client.console.HazelcastCommandLine$GlobalMixin,java.lang.String,java.lang.String,java.lang.String,java.io.File,java.util.List) throws java.io.IOException,java.lang.ClassNotFoundException,java.lang.reflect.InvocationTargetException,java.lang.IllegalAccessException): java.lang.reflect.InvocationTargetException
at picocli.CommandLine.executeUserObject(CommandLine.java:2082)
at picocli.CommandLine.access$1500(CommandLine.java:148)
at picocli.CommandLine$RunAll.recursivelyExecuteUserObject(CommandLine.java:2540)
at picocli.CommandLine$RunAll.recursivelyExecuteUserObject(CommandLine.java:2542)
at picocli.CommandLine$RunAll.handle(CommandLine.java:2536)
at picocli.CommandLine$RunAll.handle(CommandLine.java:2496)
at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:2268)
at picocli.CommandLine.parseWithHandlers(CommandLine.java:2668)
at com.hazelcast.client.console.HazelcastCommandLine.runCommandLine(HazelcastCommandLine.java:557)
at com.hazelcast.client.console.HazelcastCommandLine.main(HazelcastCommandLine.java:134)
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:118)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at com.hazelcast.instance.impl.executejar.CommandLineExecuteJar.invokeMain(CommandLineExecuteJar.java:117)
at com.hazelcast.instance.impl.executejar.CommandLineExecuteJar.executeJar(CommandLineExecuteJar.java:83)
at com.hazelcast.instance.impl.HazelcastBootstrap.executeJarOnCLI(HazelcastBootstrap.java:96)
at com.hazelcast.client.console.HazelcastCommandLine.submit(HazelcastCommandLine.java:196)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at picocli.CommandLine.executeUserObject(CommandLine.java:2070)
... 9 more
Caused by: java.lang.IllegalArgumentException: These transforms have nothing attached to them: [Watch for Changes]
at com.hazelcast.jet.impl.pipeline.Planner.validateNoLeakage(Planner.java:244)
at com.hazelcast.jet.impl.pipeline.Planner.createDag(Planner.java:85)
at com.hazelcast.jet.impl.pipeline.PipelineImpl.toDag(PipelineImpl.java:133)
at com.hazelcast.jet.impl.JobCoordinationService.lambda$submitJob$1(JobCoordinationService.java:259)
at com.hazelcast.jet.impl.JobCoordinationService.lambda$submitToCoordinatorThread$60(JobCoordinationService.java:1501)
at com.hazelcast.jet.impl.JobCoordinationService.lambda$submitToCoordinatorThread$61(JobCoordinationService.java:1522)
at com.hazelcast.internal.util.executor.CompletableFutureTask.run(CompletableFutureTask.java:64)
at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.lang.Thread.run(Thread.java:1583)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76)
at com.hazelcast.internal.util.executor.PoolExecutorThreadFactory$ManagedThread.executeRun(PoolExecutorThreadFactory.java:89)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111)
Joses-MacBook-Pro:hazelcast-image-search colljos$
OK ... so I can see from re-reading the tutorial instructions, that step 3) states:
3) Using your Java IDE, open ImagesIngestPipeline.java in the image-ingest-pipeline module. Follow the guidance and instructions in the file.
The createPipeline
method is basically incomplete.
private static Pipeline createPipeline(
String dir,
String suffix,
String wwwServer,
String pythonServiceBaseDir,
String pythonServiceModule){
Pipeline pipeline = Pipeline.create();
/*
* This custom source emits (event type, changed file name) tuples. Currently, the
* only event type returned is ADD.
*
* This source is non-distributed, which means it will run on one arbitrarily chosen node in the cluster.
*/
StreamSource<Tuple2<DirectoryWatcher.EventType, String>> directoryWatcher =
DirectoryWatcherSourceBuilder.newDirectoryWatcher(dir, suffix);
StreamStage<Tuple2<DirectoryWatcher.EventType, String>> changeEvents =
pipeline.readFrom(directoryWatcher).withoutTimestamps().setName("Watch for Changes");
/*
* TODO: The events all originate on a single server. Unless forced, subsequent processing will
* remain on the same server. In order to distribute the work, rebalance the stream.
* See: https://docs.hazelcast.org/docs/5.4.0/javadoc/com/hazelcast/jet/pipeline/StreamStage.html#rebalance()
*/
StreamStage<Tuple2<DirectoryWatcher.EventType, String>> rebalancedEvents = null;
/*
* TODO: Change the format of the input to match what is expected by the python embedding
* service. Use a "map" stage (https://docs.hazelcast.org/docs/5.4.0/javadoc/com/hazelcast/jet/pipeline/StreamStage.html#map(com.hazelcast.function.FunctionEx)
* to change the filename into a URL.
* Example: if the filename = "myimage.jpg", the URL would be wwwServer + "/" + filename
*
* Note that the input event is a Tuple2 (https://docs.hazelcast.org/docs/5.4.0/javadoc/com/hazelcast/jet/datamodel/Tuple2.html)
* The filename is in the second part of the tuple and can be accessed with tuple.f1()
*/
StreamStage<String> inputs = null;
/*
* Now use mapUsingPython to call the python embedding service.
* See the example here: https://docs.hazelcast.com/hazelcast/5.4/pipelines/python
*
* Notes: create the PythonServiceConfig with the following arguments
* baseDir: This is the name of the folder (on the client that submits the job) where the
* python service code resides. In this case, it is passed in to the createPipeline
* method as an argument.
*
* handlerModule: the name of the python file (without .py) containing the transform_list function.
* This is also passed to the createPipeline method.
*
* You can set the number of python instances to run on each node using the "setLocalParallelism"
* method of the StreamStage. Set local parallelism to 2.
*
* After creating a PythonServiceConfig, invoke the python service with code similar to
* the example below.
*
* inputs.apply(PythonTransforms.mapUsingPython(myPythonServiceConfig).setLocalParallelism(2);
*/
PythonServiceConfig pythonService = null; // create the python service config
StreamStage<String> outputs = null; // call map using python
/*
* For debugging purposes, finish the pipeline with a logging sink and then test it by deploying
* it per the instructions in the README. Remove this sink in the final pipeline.
*/
//outputs.writeTo(Sinks.logger());
/*
* Use mapUsingService to decode the output. The python stage emits json encoded strings.
* Use an instance of EmbeddingServiceCodec to translate the json into a (filename, float[]) 2-tuple.
*
* See: https://docs.hazelcast.com/hazelcast/5.4/pipelines/transforms#mapusingservice for
* an example
*
* First use ServiceFactories.sharedService to create an instance of EmbeddingServiceCodec. One instance
* will be created on each node in the cluster. Then use a mapUsingService stage to perform the decoding.
* Your map method will be passed the shared instance of EmbeddingServiceCodec and an event. You should
* call embeddingCodec.decodeOutput(myEvent)
*/
ServiceFactory<?, EmbeddingServiceCodec> postProcessor = null;
StreamStage<Tuple2<String, float[]>> vectors = null;
/*
* Finally, create a vector collection sink and write the pipeline events to it.
* See: https://docs.hazelcast.com/hazelcast/5.5-snapshot/integrate/vector-collection-connector#vector-collection-as-a-sink
*
* Call the VectorSinks.vectorCollection method with the following argument
* name of the collection (see hazelcast.yaml)
* function to extract the key - in this case, use tuple.f0() to return the URL of the image
* function to extract the value - pass the URL of the image as the value as well
* function to build an instance of VectorValue from the array of floats in tuple.f1()
*
* You can construct a VectorValue instance with VectorValues.Of(an-array-of-floats)
*
* When you have created a vector sink, finish the pipeline by writing it to the sink.
*/
Sink<Tuple2<String, float[]>> vectorCollectionSink = null;
//vectors.writeTo(vectorCollectionSink);
return pipeline;
}
This feels impossible to complete by a novice to the platform. What previous tutorials should I have completed before attempting this one? It would be useful to provide the complete solution as well (to experience running the demo without too much prior knowledge).
NB: one of the references within the code comments refer to an invalid URL (404: Page Not Found) https://docs.hazelcast.com/hazelcast/5.5-snapshot/integrate/vector-collection-connector#vector-collection-as-a-sink
Thank you for your feedback. We're looking into this.
So, two issues here. The first is docker compose run download-model
doesn't work for Jose. To address this, I've added specific versions of python and the python packages in python-container/Dockerfile
. That fixed it for me (M1 Mac) but there could be differences with platforms. We should at least test with an Intel Mac. Could someone with an Intel Mac please pull the latest and try it? Note - Docker will cache the broken image so before you retest, run docker image ls | grep hazelcast-image-search
and remove any images associated with this project with docker image rm <image-id-to-delete>
. You may have to run docker container prune
before you can delete the image. Anyway, to recap, it works for me on my m1 mac but we need someone with an Intel Mac to pull the latest and retest.
For the second issue, for those that don't want to write the image ingest pipeline themselves, there is a solution in the project here: image-ingest-pipeline/src/main/java/hazelcast/platform/labs/image/similarity/solution/ImageIngestPipelineSolution.java and there is also a task to submit the solution in the docker compose file. The solution can be run using docker compose run submit-image-loader-solution
. I think we just need to document that somewhere in the tutorial.
@wrmay Thanks for the feedback on this.
I have now applied the Solution code to the pipeline but am seeing a PythonService initialisation error upon submitting the image loader command.
Sequence of commands:
docker compose run submit-image-loader-solution
docker compose logs --follow hz
Log output of hz container:
hz-1 | 2024-11-20 11:07:54,767 [ INFO] [hz.keen_lichterman.cached.thread-21] [c.h.j.python]: [172.24.0.3]:5701 [dev] [5.5.2] Started Python process: 262
hz-1 | 2024-11-20 11:07:54,767 [ INFO] [hz.keen_lichterman.cached.thread-19] [c.h.j.python]: [172.24.0.3]:5701 [dev] [5.5.2] Started Python process: 261
hz-1 | 2024-11-20 11:08:12,915 [ERROR] [hz.keen_lichterman.cached.thread-17] [c.h.j.i.e.TaskletExecutionService]: [172.24.0.3]:5701 [dev] [5.5.2] Tasklet initialization failed
hz-1 | com.hazelcast.jet.JetException: PythonService initialization failed
hz-1 | at com.hazelcast.jet.python.PythonService.<init>(PythonService.java:78) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1 | at com.hazelcast.jet.python.PythonService.lambda$factory$f8fa7483$1(PythonService.java:91) ~[hazelcast-jet-python-5.5.2.jar:?]
I've checked the configuration and validated the expected Python file clip_image_encoder
is available on the Docker image:
The solution also didn't work for me. We may have a regression. @colljos Thanks for bringing this to our attention. This was developed against a pre-release snapshot. It worked then but it appears broken now. I'll need a little time figure out why it no longer works. If you could include the whole stack trace from the Hazelcast logs that would help me to confirm whether you are seeing the same thing as I am.
Please find attached full stack trace:
hz-1 | 2024-11-20 11:19:36,664 [ INFO] [hz.keen_lichterman.cached.thread-19] [c.h.j.python]: [172.24.0.3]:5701 [dev] [5.5.2] Started Python process: 678
hz-1 | 2024-11-20 11:19:36,664 [ INFO] [hz.keen_lichterman.cached.thread-20] [c.h.j.python]: [172.24.0.3]:5701 [dev] [5.5.2] Started Python process: 677
hz-1 | 2024-11-20 11:19:48,681 [ERROR] [hz.keen_lichterman.cached.thread-7] [c.h.j.i.e.TaskletExecutionService]: [172.24.0.3]:5701 [dev] [5.5.2] Tasklet initialization failed
hz-1 | com.hazelcast.jet.JetException: PythonService initialization failed
hz-1 | at com.hazelcast.jet.python.PythonService.<init>(PythonService.java:78) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1 | at com.hazelcast.jet.python.PythonService.lambda$factory$f8fa7483$1(PythonService.java:91) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1 | at com.hazelcast.function.BiFunctionEx.apply(BiFunctionEx.java:47) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.processor.AbstractTransformUsingServiceP.init(AbstractTransformUsingServiceP.java:44) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.processor.AsyncTransformUsingServiceOrderedP.init(AsyncTransformUsingServiceOrderedP.java:90) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.core.AbstractProcessor.init(AbstractProcessor.java:82) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.execution.ProcessorTasklet.lambda$init$2f647568$1(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.execution.ProcessorTasklet.init(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.execution.TaskletExecutionService.lambda$submitCooperativeTasklets$6(TaskletExecutionService.java:194) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
hz-1 | at com.hazelcast.internal.util.executor.CompletableFutureTask.run(CompletableFutureTask.java:64) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
hz-1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
hz-1 | at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
hz-1 | at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) [hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.internal.util.executor.PoolExecutorThreadFactory$ManagedThread.executeRun(PoolExecutorThreadFactory.java:89) [hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111) [hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | Caused by: java.io.IOException: Python process died before completing initialization
hz-1 | at com.hazelcast.jet.python.JetToPythonServer.start(JetToPythonServer.java:77) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1 | at com.hazelcast.jet.python.PythonService.<init>(PythonService.java:70) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1 | ... 21 more
hz-1 | Caused by: java.net.SocketTimeoutException: Accept timed out
hz-1 | at java.base/sun.nio.ch.NioSocketImpl.timedAccept(NioSocketImpl.java:701) ~[?:?]
hz-1 | at java.base/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:745) ~[?:?]
hz-1 | at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:698) ~[?:?]
hz-1 | at java.base/java.net.ServerSocket.platformImplAccept(ServerSocket.java:663) ~[?:?]
hz-1 | at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:639) ~[?:?]
hz-1 | at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:585) ~[?:?]
hz-1 | at java.base/java.net.ServerSocket.accept(ServerSocket.java:543) ~[?:?]
hz-1 | at com.hazelcast.jet.python.JetToPythonServer.start(JetToPythonServer.java:68) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1 | at com.hazelcast.jet.python.PythonService.<init>(PythonService.java:70) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1 | ... 21 more
hz-1 | 2024-11-20 11:19:48,684 [ERROR] [hz.keen_lichterman.cached.thread-7] [c.h.j.i.e.TaskletExecutionService]: [172.24.0.3]:5701 [dev] [5.5.2] Tasklet initialization failed
hz-1 | com.hazelcast.jet.JetException: PythonService initialization failed
hz-1 | at com.hazelcast.jet.python.PythonService.<init>(PythonService.java:78) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1 | at com.hazelcast.jet.python.PythonService.lambda$factory$f8fa7483$1(PythonService.java:91) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1 | at com.hazelcast.function.BiFunctionEx.apply(BiFunctionEx.java:47) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.processor.AbstractTransformUsingServiceP.init(AbstractTransformUsingServiceP.java:44) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.processor.AsyncTransformUsingServiceOrderedP.init(AsyncTransformUsingServiceOrderedP.java:90) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.core.AbstractProcessor.init(AbstractProcessor.java:82) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.execution.ProcessorTasklet.lambda$init$2f647568$1(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.execution.ProcessorTasklet.init(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.execution.TaskletExecutionService.lambda$submitCooperativeTasklets$6(TaskletExecutionService.java:194) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
hz-1 | at com.hazelcast.internal.util.executor.CompletableFutureTask.run(CompletableFutureTask.java:64) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
hz-1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
hz-1 | at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
hz-1 | at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) [hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.internal.util.executor.PoolExecutorThreadFactory$ManagedThread.executeRun(PoolExecutorThreadFactory.java:89) [hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111) [hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | Caused by: java.io.IOException: Python process died before completing initialization
hz-1 | at com.hazelcast.jet.python.JetToPythonServer.start(JetToPythonServer.java:77) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1 | at com.hazelcast.jet.python.PythonService.<init>(PythonService.java:70) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1 | ... 21 more
hz-1 | Caused by: java.net.SocketTimeoutException: Accept timed out
hz-1 | at java.base/sun.nio.ch.NioSocketImpl.timedAccept(NioSocketImpl.java:701) ~[?:?]
hz-1 | at java.base/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:745) ~[?:?]
hz-1 | at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:698) ~[?:?]
hz-1 | at java.base/java.net.ServerSocket.platformImplAccept(ServerSocket.java:663) ~[?:?]
hz-1 | at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:639) ~[?:?]
hz-1 | at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:585) ~[?:?]
hz-1 | at java.base/java.net.ServerSocket.accept(ServerSocket.java:543) ~[?:?]
hz-1 | at com.hazelcast.jet.python.JetToPythonServer.start(JetToPythonServer.java:68) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1 | at com.hazelcast.jet.python.PythonService.<init>(PythonService.java:70) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1 | ... 21 more
hz-1 | 2024-11-20 11:19:50,285 [ERROR] [hz.keen_lichterman.cached.thread-19] [c.h.j.i.MasterJobContext]: [172.24.0.3]:5701 [dev] [5.5.2] Execution of job '0ca6-91a3-6540-0001', execution 0ca6-91a3-d280-0001 failed
hz-1 | Start time: 2024-11-20T11:18:00.260
hz-1 | Duration: 00:01:50.025
hz-1 | To see additional job metrics enable JobConfig.storeMetricsAfterJobCompletion
hz-1 | com.hazelcast.jet.JetException: 2 of 41 tasklets failed to initialize. One of the failures is attached as the cause and its summary is com.hazelcast.jet.JetException: PythonService initialization failed
hz-1 | at com.hazelcast.jet.impl.execution.TaskletExecutionService.awaitAll(TaskletExecutionService.java:235) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.execution.TaskletExecutionService.submitCooperativeTasklets(TaskletExecutionService.java:196) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.execution.TaskletExecutionService.beginExecute(TaskletExecutionService.java:154) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.execution.ExecutionContext.beginExecution(ExecutionContext.java:233) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.JobExecutionService.beginExecution0(JobExecutionService.java:568) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.JobExecutionService.beginExecution(JobExecutionService.java:563) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.operation.StartExecutionOperation.doRun(StartExecutionOperation.java:50) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.operation.AsyncOperation.run(AsyncOperation.java:56) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:192) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:291) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:262) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:216) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.run(OperationExecutorImpl.java:459) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.runOrExecute(OperationExecutorImpl.java:519) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvokeLocal(Invocation.java:617) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:596) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke0(Invocation.java:557) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke(Invocation.java:240) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.spi.impl.operationservice.impl.InvocationBuilderImpl.invoke(InvocationBuilderImpl.java:71) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.MasterContext.invokeOnParticipant(MasterContext.java:479) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.MasterContext.invokeOnParticipants(MasterContext.java:462) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.MasterContext.invokeOnParticipants(MasterContext.java:433) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.MasterJobContext.invokeStartExecution(MasterJobContext.java:638) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.MasterJobContext.lambda$onInitStepCompleted$10(MasterJobContext.java:606) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.JobCoordinationService.lambda$submitToCoordinatorThread$60(JobCoordinationService.java:1501) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.JobCoordinationService.lambda$submitToCoordinatorThread$61(JobCoordinationService.java:1522) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.internal.util.executor.CompletableFutureTask.run(CompletableFutureTask.java:64) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
hz-1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
hz-1 | at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
hz-1 | at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) [hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.internal.util.executor.PoolExecutorThreadFactory$ManagedThread.executeRun(PoolExecutorThreadFactory.java:89) [hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111) [hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | Caused by: com.hazelcast.jet.impl.execution.TaskletExecutionException: 2 of 41 tasklets failed to initialize. One of the failures is attached as the cause and its summary is com.hazelcast.jet.JetException: PythonService initialization failed
hz-1 | at com.hazelcast.jet.impl.execution.TaskletExecutionService.awaitAll(TaskletExecutionService.java:234) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | ... 33 more
hz-1 | Caused by: com.hazelcast.jet.JetException: PythonService initialization failed
hz-1 | at com.hazelcast.jet.python.PythonService.<init>(PythonService.java:78) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1 | at com.hazelcast.jet.python.PythonService.lambda$factory$f8fa7483$1(PythonService.java:91) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1 | at com.hazelcast.function.BiFunctionEx.apply(BiFunctionEx.java:47) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.processor.AbstractTransformUsingServiceP.init(AbstractTransformUsingServiceP.java:44) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.processor.AsyncTransformUsingServiceOrderedP.init(AsyncTransformUsingServiceOrderedP.java:90) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.core.AbstractProcessor.init(AbstractProcessor.java:82) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.execution.ProcessorTasklet.lambda$init$2f647568$1(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.execution.ProcessorTasklet.init(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.execution.TaskletExecutionService.lambda$submitCooperativeTasklets$6(TaskletExecutionService.java:194) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
hz-1 | ... 8 more
hz-1 | Caused by: java.io.IOException: Python process died before completing initialization
hz-1 | at com.hazelcast.jet.python.JetToPythonServer.start(JetToPythonServer.java:77) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1 | at com.hazelcast.jet.python.PythonService.<init>(PythonService.java:70) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1 | at com.hazelcast.jet.python.PythonService.lambda$factory$f8fa7483$1(PythonService.java:91) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1 | at com.hazelcast.function.BiFunctionEx.apply(BiFunctionEx.java:47) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.processor.AbstractTransformUsingServiceP.init(AbstractTransformUsingServiceP.java:44) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.processor.AsyncTransformUsingServiceOrderedP.init(AsyncTransformUsingServiceOrderedP.java:90) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.core.AbstractProcessor.init(AbstractProcessor.java:82) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.execution.ProcessorTasklet.lambda$init$2f647568$1(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.execution.ProcessorTasklet.init(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.execution.TaskletExecutionService.lambda$submitCooperativeTasklets$6(TaskletExecutionService.java:194) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
hz-1 | ... 8 more
hz-1 | Caused by: java.net.SocketTimeoutException: Accept timed out
hz-1 | at java.base/sun.nio.ch.NioSocketImpl.timedAccept(NioSocketImpl.java:701) ~[?:?]
hz-1 | at java.base/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:745) ~[?:?]
hz-1 | at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:698) ~[?:?]
hz-1 | at java.base/java.net.ServerSocket.platformImplAccept(ServerSocket.java:663) ~[?:?]
hz-1 | at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:639) ~[?:?]
hz-1 | at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:585) ~[?:?]
hz-1 | at java.base/java.net.ServerSocket.accept(ServerSocket.java:543) ~[?:?]
hz-1 | at com.hazelcast.jet.python.JetToPythonServer.start(JetToPythonServer.java:68) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1 | at com.hazelcast.jet.python.PythonService.<init>(PythonService.java:70) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1 | at com.hazelcast.jet.python.PythonService.lambda$factory$f8fa7483$1(PythonService.java:91) ~[hazelcast-jet-python-5.5.2.jar:?]
hz-1 | at com.hazelcast.function.BiFunctionEx.apply(BiFunctionEx.java:47) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.processor.AbstractTransformUsingServiceP.init(AbstractTransformUsingServiceP.java:44) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.processor.AsyncTransformUsingServiceOrderedP.init(AsyncTransformUsingServiceOrderedP.java:90) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.core.AbstractProcessor.init(AbstractProcessor.java:82) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.execution.ProcessorTasklet.lambda$init$2f647568$1(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.execution.ProcessorTasklet.init(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at com.hazelcast.jet.impl.execution.TaskletExecutionService.lambda$submitCooperativeTasklets$6(TaskletExecutionService.java:194) ~[hazelcast-enterprise-5.5.2.jar:5.5.2]
hz-1 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
hz-1 | ... 8 more
Thank you. That confirms I'm chasing the same problem you are seeing. It may be something simple like they changed the grpc port and it's checking the wrong one. I'll get to the bottom of it.
On Wed, Nov 20, 2024 at 2:29 PM Jose Coll @.***> wrote:
Please find attached full stack trace:
hz-1 | 2024-11-20 11:19:36,664 [ INFO] [hz.keen_lichterman.cached.thread-19] [c.h.j.python]: [172.24.0.3]:5701 [dev] [5.5.2] Started Python process: 678 hz-1 | 2024-11-20 11:19:36,664 [ INFO] [hz.keen_lichterman.cached.thread-20] [c.h.j.python]: [172.24.0.3]:5701 [dev] [5.5.2] Started Python process: 677 hz-1 | 2024-11-20 11:19:48,681 [ERROR] [hz.keen_lichterman.cached.thread-7] [c.h.j.i.e.TaskletExecutionService]: [172.24.0.3]:5701 [dev] [5.5.2] Tasklet initialization failed hz-1 | com.hazelcast.jet.JetException: PythonService initialization failed hz-1 | at com.hazelcast.jet.python.PythonService.
(PythonService.java:78) ~[hazelcast-jet-python-5.5.2.jar:?] hz-1 | at com.hazelcast.jet.python.PythonService.lambda$factory$f8fa7483$1(PythonService.java:91) ~[hazelcast-jet-python-5.5.2.jar:?] hz-1 | at com.hazelcast.function.BiFunctionEx.apply(BiFunctionEx.java:47) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.processor.AbstractTransformUsingServiceP.init(AbstractTransformUsingServiceP.java:44) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.processor.AsyncTransformUsingServiceOrderedP.init(AsyncTransformUsingServiceOrderedP.java:90) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.core.AbstractProcessor.init(AbstractProcessor.java:82) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.execution.ProcessorTasklet.lambda$init$2f647568$1(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.execution.ProcessorTasklet.init(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.execution.TaskletExecutionService.lambda$submitCooperativeTasklets$6(TaskletExecutionService.java:194) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] hz-1 | at com.hazelcast.internal.util.executor.CompletableFutureTask.run(CompletableFutureTask.java:64) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] hz-1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] hz-1 | at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] hz-1 | at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) [hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.internal.util.executor.PoolExecutorThreadFactory$ManagedThread.executeRun(PoolExecutorThreadFactory.java:89) [hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111) [hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | Caused by: java.io.IOException: Python process died before completing initialization hz-1 | at com.hazelcast.jet.python.JetToPythonServer.start(JetToPythonServer.java:77) ~[hazelcast-jet-python-5.5.2.jar:?] hz-1 | at com.hazelcast.jet.python.PythonService. (PythonService.java:70) ~[hazelcast-jet-python-5.5.2.jar:?] hz-1 | ... 21 more hz-1 | Caused by: java.net.SocketTimeoutException: Accept timed out hz-1 | at java.base/sun.nio.ch.NioSocketImpl.timedAccept(NioSocketImpl.java:701) ~[?:?] hz-1 | at java.base/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:745) ~[?:?] hz-1 | at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:698) ~[?:?] hz-1 | at java.base/java.net.ServerSocket.platformImplAccept(ServerSocket.java:663) ~[?:?] hz-1 | at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:639) ~[?:?] hz-1 | at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:585) ~[?:?] hz-1 | at java.base/java.net.ServerSocket.accept(ServerSocket.java:543) ~[?:?] hz-1 | at com.hazelcast.jet.python.JetToPythonServer.start(JetToPythonServer.java:68) ~[hazelcast-jet-python-5.5.2.jar:?] hz-1 | at com.hazelcast.jet.python.PythonService. (PythonService.java:70) ~[hazelcast-jet-python-5.5.2.jar:?] hz-1 | ... 21 more hz-1 | 2024-11-20 11:19:48,684 [ERROR] [hz.keen_lichterman.cached.thread-7] [c.h.j.i.e.TaskletExecutionService]: [172.24.0.3]:5701 [dev] [5.5.2] Tasklet initialization failed hz-1 | com.hazelcast.jet.JetException: PythonService initialization failed hz-1 | at com.hazelcast.jet.python.PythonService. (PythonService.java:78) ~[hazelcast-jet-python-5.5.2.jar:?] hz-1 | at com.hazelcast.jet.python.PythonService.lambda$factory$f8fa7483$1(PythonService.java:91) ~[hazelcast-jet-python-5.5.2.jar:?] hz-1 | at com.hazelcast.function.BiFunctionEx.apply(BiFunctionEx.java:47) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.processor.AbstractTransformUsingServiceP.init(AbstractTransformUsingServiceP.java:44) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.processor.AsyncTransformUsingServiceOrderedP.init(AsyncTransformUsingServiceOrderedP.java:90) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.core.AbstractProcessor.init(AbstractProcessor.java:82) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.execution.ProcessorTasklet.lambda$init$2f647568$1(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.execution.ProcessorTasklet.init(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.execution.TaskletExecutionService.lambda$submitCooperativeTasklets$6(TaskletExecutionService.java:194) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] hz-1 | at com.hazelcast.internal.util.executor.CompletableFutureTask.run(CompletableFutureTask.java:64) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] hz-1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] hz-1 | at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] hz-1 | at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) [hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.internal.util.executor.PoolExecutorThreadFactory$ManagedThread.executeRun(PoolExecutorThreadFactory.java:89) [hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111) [hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | Caused by: java.io.IOException: Python process died before completing initialization hz-1 | at com.hazelcast.jet.python.JetToPythonServer.start(JetToPythonServer.java:77) ~[hazelcast-jet-python-5.5.2.jar:?] hz-1 | at com.hazelcast.jet.python.PythonService. (PythonService.java:70) ~[hazelcast-jet-python-5.5.2.jar:?] hz-1 | ... 21 more hz-1 | Caused by: java.net.SocketTimeoutException: Accept timed out hz-1 | at java.base/sun.nio.ch.NioSocketImpl.timedAccept(NioSocketImpl.java:701) ~[?:?] hz-1 | at java.base/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:745) ~[?:?] hz-1 | at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:698) ~[?:?] hz-1 | at java.base/java.net.ServerSocket.platformImplAccept(ServerSocket.java:663) ~[?:?] hz-1 | at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:639) ~[?:?] hz-1 | at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:585) ~[?:?] hz-1 | at java.base/java.net.ServerSocket.accept(ServerSocket.java:543) ~[?:?] hz-1 | at com.hazelcast.jet.python.JetToPythonServer.start(JetToPythonServer.java:68) ~[hazelcast-jet-python-5.5.2.jar:?] hz-1 | at com.hazelcast.jet.python.PythonService. (PythonService.java:70) ~[hazelcast-jet-python-5.5.2.jar:?] hz-1 | ... 21 more hz-1 | 2024-11-20 11:19:50,285 [ERROR] [hz.keen_lichterman.cached.thread-19] [c.h.j.i.MasterJobContext]: [172.24.0.3]:5701 [dev] [5.5.2] Execution of job '0ca6-91a3-6540-0001', execution 0ca6-91a3-d280-0001 failed hz-1 | Start time: 2024-11-20T11:18:00.260 hz-1 | Duration: 00:01:50.025 hz-1 | To see additional job metrics enable JobConfig.storeMetricsAfterJobCompletion hz-1 | com.hazelcast.jet.JetException: 2 of 41 tasklets failed to initialize. One of the failures is attached as the cause and its summary is com.hazelcast.jet.JetException: PythonService initialization failed hz-1 | at com.hazelcast.jet.impl.execution.TaskletExecutionService.awaitAll(TaskletExecutionService.java:235) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.execution.TaskletExecutionService.submitCooperativeTasklets(TaskletExecutionService.java:196) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.execution.TaskletExecutionService.beginExecute(TaskletExecutionService.java:154) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.execution.ExecutionContext.beginExecution(ExecutionContext.java:233) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.JobExecutionService.beginExecution0(JobExecutionService.java:568) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.JobExecutionService.beginExecution(JobExecutionService.java:563) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.operation.StartExecutionOperation.doRun(StartExecutionOperation.java:50) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.operation.AsyncOperation.run(AsyncOperation.java:56) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.spi.impl.operationservice.Operation.call(Operation.java:192) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.call(OperationRunnerImpl.java:291) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:262) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:216) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.run(OperationExecutorImpl.java:459) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl.runOrExecute(OperationExecutorImpl.java:519) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvokeLocal(Invocation.java:617) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.spi.impl.operationservice.impl.Invocation.doInvoke(Invocation.java:596) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke0(Invocation.java:557) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.spi.impl.operationservice.impl.Invocation.invoke(Invocation.java:240) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.spi.impl.operationservice.impl.InvocationBuilderImpl.invoke(InvocationBuilderImpl.java:71) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.MasterContext.invokeOnParticipant(MasterContext.java:479) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.MasterContext.invokeOnParticipants(MasterContext.java:462) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.MasterContext.invokeOnParticipants(MasterContext.java:433) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.MasterJobContext.invokeStartExecution(MasterJobContext.java:638) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.MasterJobContext.lambda$onInitStepCompleted$10(MasterJobContext.java:606) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.JobCoordinationService.lambda$submitToCoordinatorThread$60(JobCoordinationService.java:1501) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.JobCoordinationService.lambda$submitToCoordinatorThread$61(JobCoordinationService.java:1522) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.internal.util.executor.CompletableFutureTask.run(CompletableFutureTask.java:64) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.internal.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:217) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] hz-1 | at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] hz-1 | at java.base/java.lang.Thread.run(Thread.java:1583) [?:?] hz-1 | at com.hazelcast.internal.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:76) [hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.internal.util.executor.PoolExecutorThreadFactory$ManagedThread.executeRun(PoolExecutorThreadFactory.java:89) [hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111) [hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | Caused by: com.hazelcast.jet.impl.execution.TaskletExecutionException: 2 of 41 tasklets failed to initialize. One of the failures is attached as the cause and its summary is com.hazelcast.jet.JetException: PythonService initialization failed hz-1 | at com.hazelcast.jet.impl.execution.TaskletExecutionService.awaitAll(TaskletExecutionService.java:234) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | ... 33 more hz-1 | Caused by: com.hazelcast.jet.JetException: PythonService initialization failed hz-1 | at com.hazelcast.jet.python.PythonService. (PythonService.java:78) ~[hazelcast-jet-python-5.5.2.jar:?] hz-1 | at com.hazelcast.jet.python.PythonService.lambda$factory$f8fa7483$1(PythonService.java:91) ~[hazelcast-jet-python-5.5.2.jar:?] hz-1 | at com.hazelcast.function.BiFunctionEx.apply(BiFunctionEx.java:47) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.processor.AbstractTransformUsingServiceP.init(AbstractTransformUsingServiceP.java:44) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.processor.AsyncTransformUsingServiceOrderedP.init(AsyncTransformUsingServiceOrderedP.java:90) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.core.AbstractProcessor.init(AbstractProcessor.java:82) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.execution.ProcessorTasklet.lambda$init$2f647568$1(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.execution.ProcessorTasklet.init(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.execution.TaskletExecutionService.lambda$submitCooperativeTasklets$6(TaskletExecutionService.java:194) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] hz-1 | ... 8 more hz-1 | Caused by: java.io.IOException: Python process died before completing initialization hz-1 | at com.hazelcast.jet.python.JetToPythonServer.start(JetToPythonServer.java:77) ~[hazelcast-jet-python-5.5.2.jar:?] hz-1 | at com.hazelcast.jet.python.PythonService. (PythonService.java:70) ~[hazelcast-jet-python-5.5.2.jar:?] hz-1 | at com.hazelcast.jet.python.PythonService.lambda$factory$f8fa7483$1(PythonService.java:91) ~[hazelcast-jet-python-5.5.2.jar:?] hz-1 | at com.hazelcast.function.BiFunctionEx.apply(BiFunctionEx.java:47) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.processor.AbstractTransformUsingServiceP.init(AbstractTransformUsingServiceP.java:44) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.processor.AsyncTransformUsingServiceOrderedP.init(AsyncTransformUsingServiceOrderedP.java:90) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.core.AbstractProcessor.init(AbstractProcessor.java:82) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.execution.ProcessorTasklet.lambda$init$2f647568$1(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.execution.ProcessorTasklet.init(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.execution.TaskletExecutionService.lambda$submitCooperativeTasklets$6(TaskletExecutionService.java:194) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] hz-1 | ... 8 more hz-1 | Caused by: java.net.SocketTimeoutException: Accept timed out hz-1 | at java.base/sun.nio.ch.NioSocketImpl.timedAccept(NioSocketImpl.java:701) ~[?:?] hz-1 | at java.base/sun.nio.ch.NioSocketImpl.accept(NioSocketImpl.java:745) ~[?:?] hz-1 | at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:698) ~[?:?] hz-1 | at java.base/java.net.ServerSocket.platformImplAccept(ServerSocket.java:663) ~[?:?] hz-1 | at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:639) ~[?:?] hz-1 | at java.base/java.net.ServerSocket.implAccept(ServerSocket.java:585) ~[?:?] hz-1 | at java.base/java.net.ServerSocket.accept(ServerSocket.java:543) ~[?:?] hz-1 | at com.hazelcast.jet.python.JetToPythonServer.start(JetToPythonServer.java:68) ~[hazelcast-jet-python-5.5.2.jar:?] hz-1 | at com.hazelcast.jet.python.PythonService. (PythonService.java:70) ~[hazelcast-jet-python-5.5.2.jar:?] hz-1 | at com.hazelcast.jet.python.PythonService.lambda$factory$f8fa7483$1(PythonService.java:91) ~[hazelcast-jet-python-5.5.2.jar:?] hz-1 | at com.hazelcast.function.BiFunctionEx.apply(BiFunctionEx.java:47) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.processor.AbstractTransformUsingServiceP.init(AbstractTransformUsingServiceP.java:44) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.processor.AsyncTransformUsingServiceOrderedP.init(AsyncTransformUsingServiceOrderedP.java:90) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.core.AbstractProcessor.init(AbstractProcessor.java:82) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.execution.ProcessorTasklet.lambda$init$2f647568$1(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.execution.ProcessorTasklet.init(ProcessorTasklet.java:279) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.function.RunnableEx.run(RunnableEx.java:31) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.util.Util.doWithClassLoader(Util.java:586) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at com.hazelcast.jet.impl.execution.TaskletExecutionService.lambda$submitCooperativeTasklets$6(TaskletExecutionService.java:194) ~[hazelcast-enterprise-5.5.2.jar:5.5.2] hz-1 | at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?] hz-1 | ... 8 more — Reply to this email directly, view it on GitHub https://github.com/hazelcast/hz-docs/issues/1350#issuecomment-2489385317, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAV3MN53UAQ34MWHU4TBBXD2BTPI3AVCNFSM6AAAAABQTCEA66VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBZGM4DKMZRG4 . You are receiving this because you were mentioned.Message ID: @.***>
-- Randy May Industry Solutions Advocate @.*** [image: Hazelcast] https://hazelcast.com/?utm_source=email&utm_medium=signature&utm_campaign=email-signature | Palo Alto, CA 94306 | USA +1 404.414.5730 | hazelcast.com https://www.hazelcast.com/?utm_source=email&utm_medium=signature&utm_campaign=email-signature
-- This message contains confidential information and is intended only for the individuals named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required, please request a hard-copy version. -Hazelcast
Hi, I have some feedback about this page
Tutorial Setup step 2 continuously fails:
Error output:
Here is my environment: