We need a dt, created, modified in label maker

[laszewsk]

The logic for time reporting in the workflow needs to change

obviouslye we need to have

• now.%m/%d/%Y, %H:%M:%S: replaces time with given format

we also need

• created.%m/%d/%Y, %H:%M:%S time when the node was created
• modified.%m/%d/%Y, %H:%M:%S time when the node was modified
• dt.%m/%d/%Y, %H:%M:%S time between a start time passed to labelmaker(typically the first node) and the current time. dt is frozen after the progress is 100 and job is done.

There is the possibility to just ad a dt to the workflow node just as we have created and modified, This way it is easy to freeze it as we will not change it after the progress is 100

I think this is easy to cahnge as we can just pass all values of a node into labelmaker and than through labels bick the right values to replace

We could ven add a flag such as relative=True and all times are reported relative to t0

THis is not completed and possibly now not relevant as we also need created and modified

we need a dt in labelmaker, we can se cloudmesh variables to record the time of a start of a workflow such as

dt.name where name is the name of the workflow

so dt may look like

{dt.name.%m/%d/%Y, %H:%M:%S}

where name is the name of the workflow

t0 is then used in variables as

t0_name

dt then parses the name and format and can tahn use

t0 = variables[f"t0_{name}"]

• [x] doc: provide documentation on how to generate workflows, inclusing labelmaker

[laszewsk]

some add ons to the code have already been made, but t0 needs to be implemented.

[jpfleischer]

created has been implemented. https://github.com/cloudmesh/cloudmesh-cc/commit/09251ebda704f9490a1db90bd146f33dc0e2791a

dt is finicky because it is not built on a job-by-job basis. it only applies to whole workflow. furthermore, there is a bug where running the same workflow will build upon the same dt as the previous one, and not reset it.

this is because there is no way for labelmaker.py to know whether this is a new run or just a continuation of a workflow.

you can test by running pytest -v -x --capture=no tests/test_080_workflow_clean.py

[jpfleischer]

i just implemented a fix for dt. https://github.com/cloudmesh/cloudmesh-cc/commit/362fc66468ac9693c237e190cc2551431a9d7b21

the same issue applies, however, that dt is not job-by-job, but only applies to the whole workflow. however, it resets when it is a new run of workflow.

you can test by running pytest -v -x --capture=no tests/test_080_workflow_clean.py

[jpfleischer]

notice that, currently, this code is in the test:

    created_time = w.created_time
    ....
    label = "{{name}}\\nnow={{now.%m/%d/%Y, %H:%M:%S}}\\nprogress={{progress}}\\ncreated={created_time}\\n{{dt.workflow-clean_%H:%M:%S}} sec elapsed"
    label = label.format(created_time=created_time)

this method is a little bit messy, but it interfaces with the workflow more nicely as it takes the time directly from the instance of the workflow, and writes it into the label variable before sending it off to labelmaker.

there is another way:

label = "{name}\\nnow={now.%m/%d/%Y, %H:%M:%S}\\nprogress={progress}\\ncreated={cm.created_time_workflow-clean}\\n{dt.workflow-clean_%H:%M:%S} sec elapsed"

notice how this version uses "cm.created_time_workflow-clean" instead, which is using the cm. method, which looks at Variables() as a part of cms set. this way does not require the double braces or the .format before sending it off to labelmaker, but unfortunately this method requires the workflow name to be hardcoded in: "cm.createdtimeworkflow-clean".

which way do you think is better?

[laszewsk]

Based on your comments I thought about this a bit and here some more thoughts:

requirement: Please note that we must support running multiple workflows. THus it is best to integrate that info into the execution of the workflow not using cm variables

Also we need to identify if the timse in the execution Is actually the created time at the beginning of the time when it is created first and not overwritten (We hacked this a bit I believe) We have two files

a) a spec file so we can recreate a new workflow b) a log file that executes the workflow after initiating it from the spec

Note that any modification of progress will need to update the modified field and is always in the log file

Originally we were supposed to use yamldb for storing the workflow as log, but that was not implemented, so instead we run our own storage ... which is ok. but I have not checked if the ct\reated or modified times are actually correct,.

As they are part of the workflow we should use them from the nodes directly

As for the t0 we can simply add this to the workflow as additional field. This would I think be the cleanest implementation.

I would first start looking at a) created and modified b) integrate start time (t0) you set that in toporun

THinking this over we also need a start time for each node.

So t0 could be min (star times for al nodes) but better to have a global variable in the workflow

I suggest to add a new field such as

run: start: the starttime in DataTime format end: the endtime in DataTime format

[laszewsk]

so we should make sure we have the following in the init method

Workflow.init(specification=None, log=None, clear=False) Workflow.update() reads in the status from the various remote locations the workflow is using

clear= False, reads in the log file

if specification is None: read the log

if specification and log is None then Console.error

[laszewsk]

currently we have

def init(self, name="graph", filename=None, clear=True):

which is great as name specifies the log file ...

I forgot what filename actually means is ist the specification?

[jpfleischer]

filename tries to load the yaml file, so if the specification means the yaml file, then yes.

[laszewsk]

has this been completed?

[jpfleischer]

@laszewsk we cannot go further because the .format method in labelmaker does not allow colons. https://stackoverflow.com/questions/19913504/how-do-i-escape-a-colon-in-python-format-when-using-kwargs

so, we cannot have labels with colons in them.. like if we wanted to show the time, like 10:55 AM.. it is not possible. https://github.com/cloudmesh/cloudmesh-cc/blob/0df34fa3bf205cb36aae8d0606874421c38dbb8b/cloudmesh/cc/labelmaker.py#L226

[jpfleischer]

done https://cloudmesh.github.io/cloudmesh-cc/specifying-workflows.html#defining-nodes-in-the-workflow