Open tdoan2010 opened 1 year ago
Processing request message example:
# Generated fields
processor-name: "ocrd-.*" # e.g., "ocrd-cis-ocropy-binarize", not necessary for processing, but might be useful for human
job_id: "uuid" # e.g., "fe869a65-ea1c-4be5-8053-a3b8e01beecd", generated by the Processing Broker
# Mandatory fields - provided by the User/Workflow Server to the Processing Broker
input_file_grp: "OCR-D-.*" # e.g., "OCR-D-DEFAULT"
workspace: # either of the two fields must be available
id: "uuid" # e.g., "4a5795d6-136c-40b7-8eed-eeca8ff1c249", generated by the Workspace Server
path_to_mets: "absolute path" # e.g., "/workspaces/workspace1/mets.xml", locally available on the Processing Worker
# Optional fields - provided by the User/Workflow Server to the Processing Broker
output_file_grp: "OCR-D-.*" # e.g., "OCR-D-BIN"
page_id: "id" # e.g., "PHYS_0005..PHYS_0010" will process only pages between 5-10
overwrite: boolean
parameter:
- key_1: value
- ..
- key_N: value
log_level: "level" # e.g., "INFO"
# "save the resource usage of the processor to a file in the root directory"
save_profiling_to: "path" # e.g., "./logs/resource-usage.txt"
# If this field is not provided, do not push results back to the message queue
result_queue: "name" # e.g., "ocrd-cis-ocropy-binarize-result"
Result response message example:
# Mandatory fields - reused from the processing request message or generated by the Processing Worker
job_id: "uuid" # e.g., "fe869a65-ea1c-4be5-8053-a3b8e01beecd"
status: "value" # e.g., completed/failed
# Optional fields - available only if provided in the processing request message
path_to_mets: "absolute path"
EDIT: This post gets updated and contains the latest version of the message drafts.
Sorry, I find it a bit difficult to read your draft. What syntax is it? Could you just make one or some message examples?
My first comment is: why do we need to include message_type
in the message? Different types will be pushed to different queues. So for me, message_type
is not necessary.
why do we need to include message_type in the message? Different types will be pushed to different queues. So for me, message_type is not necessary.
It just represents the two message types, it's not a field. Okay, let me edit that and replace the regex patterns with examples.
processing message type:
workspace_ref: &reference
ocrd_processor_name: ocrd-cis-ocropy-binarize
parameters:
input_filegrp: OCR-D-MIN
output_filegrp: OCR-D-BIN
page_id: PHYS0010-PHYS0015
other_paramters: {}
push_b: true
MQ_res: &queueName
retries: 3
job_id: 1024 (auto generated)
timestamp: 20221118_153107 (auto generated)
message_hash: {} (auto generated based on the fields above)
result message type:
job_id: 1024
job_status: completed
return_code: 0
err_ref: &reference
log_ref: &reference
I would reuse the structure of the POST request to the processing endpoint of the Web API as much as possible. So, I suggest a message structure like this:
# This is not necessary for processing, but might be useful for human
processor_name: ocrd-cis-ocropy-binarize
path: /path/to/mets.xml
description: some text description here
input_file_grps:
- OCR-D-INPUT
output_file_grps:
- OCR-D-OUTPUT
page_id: PHYS_001,PHYS_002
parameters:
params_1: 1
params_2: 2
# If this field is not provided, do not push results back to the message queue
result_queue_name: ocrd-cis-ocropy-binarize-result
# The time this message is created in Unix time
created_time: 1668782988590
The job_id
should be the ID of the document in MongoDB. When a processing server gets a message, it should first create an entry in the MongoDB (with status running
) and use the ID of that entry as job_id
for the result message (if a result message is required).
job_id
, yes, that is what I meant with the (auto-generated)
it will be a uuid
generated id. The way in which we currently generate ids for workspace and workflow uploads.After rechecking the specs and reconsidering some things in more depth, here is an update that, hopefully, covers all possible use cases. I will edit my first post and pin the examples to the top if there are no current objections. @tdoan2010
Processing request message example:
# Generated fields - automatically generated by the Processing Broker
# This is not necessary for processing, but might be useful for human
processor-name: "ocrd-.*" # e.g., "ocrd-cis-ocropy-binarize"
job-id: "uuid" # e.g., "fe869a65-ea1c-4be5-8053-a3b8e01beecd"
# Mandatory fields - provided by the User/Workflow Server to the Processing Broker
workspace:
# Generated by the Workspace Server when a workspace is posted
id: "uuid" # e.g., "4a5795d6-136c-40b7-8eed-eeca8ff1c249"
input-file-grp: "OCR-D-.*" # e.g., "OCR-D-DEFAULT"
# Optional fields - provided by the User/Workflow Server to the Processing Broker
output-file-grp: "OCR-D-.*" # e.g., "OCR-D-BIN"
page-id: "id" # e.g., "PHYS_0005..PHYS_0010" will process only pages between 5-10
overwrite: boolean
parameter:
- key-1: value
- ..
- key-N: value
mets: "path" # e.g., "uuid/mets/mets.xml"
log-level: "level" # e.g., "INFO"
# "save the resource usage of the processor to a file in the root directory"
show-resource: "path" # e.g., "uuid/logs/resource-usage.txt"
# If this field is not provided, do not push results back to the message queue
result-queue: "name" # e.g., "ocrd-cis-ocropy-binarize-result"
Result response message example:
# Generated previously by the Processing Broker
job-id: "uuid" # e.g., "fe869a65-ea1c-4be5-8053-a3b8e01beecd"
# Generated by the Processing Worker
status: "value" # e.g., completed/failed
return: "value" # e.g., 0/-1
log-output: "output text"
Notes:
discovery
endpoint of the WebAPI, i.e., -J
, -L
, and -h
.id
instead of a path for workspace
. The workspace id is unique and can also contain valuable data. Currently, we generate a random UUID (type uuid4), so no valuable data can be extracted. We could consider using another type, e.g., uuid1
. Check here for more details. In that case, we would not need the created_time
field and be able to extract that from the uuid1
.description
field of the workspace
. If needed, this information could be extracted by the Processing Worker
from the workspace (when imported from the NFS). Let's keep the message size as small/compact as possible.Processing Worker
will be its id - UUID
. The mets
path must point somewhere inside the root directory.Some comments from my side:
job_id
instead of job-id
.id
nested under workspace
? Can we just have 1 field called workspace_id
?show-resource
should be named save_profiling_to
.workspace_id
over path_to_mets
. Imagine the case when users already have a lot of data stored in their hard drive. If workspace_id
is required, then they must somehow register those workspaces to obtain their IDs. So maybe a new feature, register workspace, of the Workspace Server
is needed. Plus, by using workspace_id
, the Processing Server has to first resolve those IDs, which leads to additional requests to make.workspace_id
, I would suggest supporting both: eitherworkspace:
id: 1234
path_to_mets: relative/path/to/mets.xml
or only path_to_mets
with absolute path.
log_output
in the result message because it is too much. Simply write the log directly to the database.job_id
, path_to_mets
, and status
. And they all are at the same level, no nesting.
- Please use underscore to separate words instead of dash, e.g.
job_id
instead ofjob-id
.
As long as we use the same convention everywhere, i.e. the dash, I am okay with that.
- Why do you have
id
nested underworkspace
? Can we just have 1 field calledworkspace_id
?
I left it nested after removing the description
field and assuming that we may put other fields there.
workspace_id: "uuid"
is also fine.
show-resource
should be namedsave_profiling_to
Agree
- I don't see the benefit of using
workspace_id
overpath_to_mets
. Imagine the case when users already have a lot of data stored in their hard drive. Ifworkspace_id
is required, then they must somehow register those workspaces to obtain their IDs. So maybe a new feature, register workspace, of theWorkspace Server
is needed. Plus, by usingworkspace_id
, the Processing Server has to first resolve those IDs, which leads to additional requests to make.
Good point. I have assumed that the workspaces will be stored on external storage. My conceptual understanding was that each Processing Worker
will be a separate ocr-d processor. In that case, it did not make sense to think about the local storage case. Especially, when the diagram points from the Processing Workers to a Network File System. After checking #137, I see that different kinds of ocr-d processors may be deployed on the same host.
- I would suggest supporting both: either or
path_to_mets
with absolute path.
I think supporting both and making just one of the fields mandatory is the right approach to staying flexible with our options. Btw, there is a mets
field as well. This field could be used for providing local paths. We just have to make the workspace_id
field optional as well. Let me know if that approach is better?
I think utilizing the workspace_id
field makes more sense for better scaling when using external storage and providing the OCR-D processors as a service. I imagine a group of users that will just upload their workspaces+workflows and gather the results. In that case, we stay more flexible on how and where to store the data without being stuck with a single path. What is your opinion on that?
- I would not include
log_output
in the result message because it is too much. Simply write the log directly to the database.
Agree
- In the result message, I would include only job_id, path_to_mets, and status. And they all are at the same level, no nesting.
Okay
I will edit my first post on top with the suggested changes so far and it will be the most up-to-date version.
Good point. I have assumed that the workspaces will be stored on external storage. My conceptual understanding was that each
Processing Worker
will be a separate ocr-d processor. In that case, it did not make sense to think about the local storage case. Especially, when the diagram points from the Processing Workers to a Network File System. After checking https://github.com/OCR-D/zenhub/issues/137, I see that different kinds of ocr-d processors may be deployed on the same host.
External storage can just be mounted. That's why there is NFS in the picture. The idea is: all processing servers share the same storage, either via local mount or network mount.
I think supporting both and making just one of the fields mandatory is the right approach to staying flexible with our options. Btw, there is a
mets
field as well. This field could be used for providing local paths. We just have to make the workspace_id field optional as well. Let me know if that approach is better?
Yes, either workspace
or mets
must be presented, not both, not none. This approach will also work when we have OCR-as-a-Service, in case users need to upload (or pull) data to the system.
PS: in the future, please do not edit your message completely as you did, but instead posting new ones. It would make it difficult for people (myself included) to follow the discussion when the old messages were edited.
The idea is: all processing servers share the same storage, either via local mount or network mount.
Okay, now I clearly see why providing the path is a cleaner approach when the storage is shared among the processing workers.
PS: in the future, please do not edit your message completely as you did, but instead posting new ones. It would make it difficult for people (myself included) to follow the discussion when the old messages were edited.
The first draft was compact but super complicated for others to understand. I preferred to get rid of that.
I do not agree with your current suggestion yet. Some points from my side:
id
under workspace
is cumbersome. We just need to have a field called workspace_id
, and it's mutual exclusive with path_to_mets
.log_level
is defined when we start a Processing Server, not for every run, or? Can we change the log level of a running processor?save_profiling_to
. The profile option must be provided at the time we start a processor, and we cannot change it when the processor is running, or?input-file-grp
and output-file-grp
must be lists, not single value.overwrite
value of a running processor, right?So, my suggested example:
job_id: uuid
processor_name: ocrd-cis-ocropy-binarize
path_to_mets: /path/to/mets.xml
input_file_grps:
- OCR-D-INPUT
output_file_grps:
- OCR-D-OUTPUT
page_id: PHYS_001,PHYS_002
parameters:
params_1: 1
params_2: 2
# If this field is not provided, do not push results back to the message queue
result_queue_name: ocrd-cis-ocropy-binarize-result
# The time this message is created in Unix time
created_time: 1668782988590
Except job_id
, processor_name
, and created_time
, the rest comes from the body of the POST /processor/{executable}
request.
- Nesting id under workspace is cumbersome. We just need to have a field called workspace_id, and it's mutual exclusive with path_to_mets.
Agree.
- log_level is defined when we start a Processing Server, not for every run, or? Can we change the log level of a running processor?
Having more flexibility may help here. We could change the log level without having to restart the processor. Currently, I think it's not possible to change the log level of a running processor. However, this may be achievable if the log level is defined with an environment variable. We could drop that field out for now.
- Same with save_profiling_to.
Yeah, the same as in 2. We could drop that too.
- input-file-grp and output-file-grp must be lists, not single value.
I decided to go with a single value to avoid having to parse the input/output. I could change that to a list but then extra work has to be done to adapt the input/output to the processor parameters for each processing request. EDIT: Extra work = Extra processing time.
- We cannot change the overwrite value of a running processor, right?
Yes, same as in 2 and 3.
For input-file-grp
and output-file-grp
, please keep them as lists. "Extra work" is just 1 line Python code to convert a list into a comma separated string :)
Okay, here is the newer version with the proposed changes.
Processing request message example:
# Generated fields
processor-name: "ocrd-.*" # e.g., "ocrd-cis-ocropy-binarize", not necessary for processing, but might be useful for human
job_id: "uuid" # e.g., "fe869a65-ea1c-4be5-8053-a3b8e01beecd", generated by the Processing Broker
# Mandatory fields - provided by the User/Workflow Server to the Processing Broker
input_file_grp:
- "OCR-D-.*" # e.g., "OCR-D-DEFAULT"
# either of the two fields below must be available
workspace_id: "uuid" # e.g., "4a5795d6-136c-40b7-8eed-eeca8ff1c249", generated by the Workspace Server
path_to_mets: "absolute path" # e.g., "/workspaces/workspace1/mets.xml", locally available on the Processing Worker
# Optional fields - provided by the User/Workflow Server to the Processing Broker
output_file_grp:
- "OCR-D-.*" # e.g., "OCR-D-BIN"
page_id: "id" # e.g., "PHYS_0005..PHYS_0010" will process only pages between 5-10
parameters:
- params_1: "value"
- ..
- params_N: "value"
# If this field is not provided, do not push results back to the message queue
result_queue_name: "name" # e.g., "ocrd-cis-ocropy-binarize-result"
Result response message example:
# Mandatory fields - reused from the processing request message or generated by the Processing Worker
job_id: "uuid" # e.g., "fe869a65-ea1c-4be5-8053-a3b8e01beecd"
status: "value" # e.g., completed/failed
# Optional fields - available only if provided in the processing request message
path_to_mets: "absolute path"
For input-file-grp and output-file-grp, please keep them as lists. "Extra work" is just 1 line Python code to convert a list into a comma separated string :)
By "extra work" I meant the conversion that will potentially happen millions of times (depending on how many messages are there) when parsing the messages, not that it's extra work for us to implement.
@MehmedGIT I created 2 JSON schemas, one for processing messages and one for result messages. You can find them currently in https://github.com/OCR-D/spec/pull/222. Please check them and try using them in your program to validate processing message. Some notes from me:
created_time
is required in processing messages.path_to_mets
or workspace_id
must be presented in result messages.I will have a look and implement them soon.
We need to come up with a schema for the message which submit to the queue. There might be different type of message, e.g. processing, result, error, logging, etc.
Use the ocrd_tool.schema.yml as a reference implementation to write this schema.