Closed laszewsk closed 1 year ago
some add ons to the code have already been made, but t0 needs to be implemented.
created has been implemented. https://github.com/cloudmesh/cloudmesh-cc/commit/09251ebda704f9490a1db90bd146f33dc0e2791a
dt is finicky because it is not built on a job-by-job basis. it only applies to whole workflow. furthermore, there is a bug where running the same workflow will build upon the same dt as the previous one, and not reset it.
this is because there is no way for labelmaker.py to know whether this is a new run or just a continuation of a workflow.
you can test by running pytest -v -x --capture=no tests/test_080_workflow_clean.py
i just implemented a fix for dt. https://github.com/cloudmesh/cloudmesh-cc/commit/362fc66468ac9693c237e190cc2551431a9d7b21
the same issue applies, however, that dt is not job-by-job, but only applies to the whole workflow. however, it resets when it is a new run of workflow.
you can test by running pytest -v -x --capture=no tests/test_080_workflow_clean.py
notice that, currently, this code is in the test:
created_time = w.created_time
....
label = "{{name}}\\nnow={{now.%m/%d/%Y, %H:%M:%S}}\\nprogress={{progress}}\\ncreated={created_time}\\n{{dt.workflow-clean_%H:%M:%S}} sec elapsed"
label = label.format(created_time=created_time)
this method is a little bit messy, but it interfaces with the workflow more nicely as it takes the time directly from the instance of the workflow, and writes it into the label variable before sending it off to labelmaker.
there is another way:
label = "{name}\\nnow={now.%m/%d/%Y, %H:%M:%S}\\nprogress={progress}\\ncreated={cm.created_time_workflow-clean}\\n{dt.workflow-clean_%H:%M:%S} sec elapsed"
notice how this version uses "cm.created_time_workflow-clean" instead, which is using the cm. method, which looks at Variables() as a part of cms set. this way does not require the double braces or the .format before sending it off to labelmaker, but unfortunately this method requires the workflow name to be hardcoded in: "cm.createdtimeworkflow-clean".
which way do you think is better?
Based on your comments I thought about this a bit and here some more thoughts:
requirement: Please note that we must support running multiple workflows. THus it is best to integrate that info into the execution of the workflow not using cm variables
Also we need to identify if the timse in the execution Is actually the created time at the beginning of the time when it is created first and not overwritten (We hacked this a bit I believe) We have two files
a) a spec file so we can recreate a new workflow b) a log file that executes the workflow after initiating it from the spec
Note that any modification of progress will need to update the modified field and is always in the log file
Originally we were supposed to use yamldb for storing the workflow as log, but that was not implemented, so instead we run our own storage ... which is ok. but I have not checked if the ct\reated or modified times are actually correct,.
As they are part of the workflow we should use them from the nodes directly
As for the t0 we can simply add this to the workflow as additional field. This would I think be the cleanest implementation.
I would first start looking at a) created and modified b) integrate start time (t0) you set that in toporun
THinking this over we also need a start time for each node.
So t0 could be min (star times for al nodes) but better to have a global variable in the workflow
I suggest to add a new field such as
run: start: the starttime in DataTime format end: the endtime in DataTime format
so we should make sure we have the following in the init method
Workflow.init(specification=None, log=None, clear=False) Workflow.update() reads in the status from the various remote locations the workflow is using
clear= False, reads in the log file
if specification is None: read the log
if specification and log is None then Console.error
currently we have
def init(self, name="graph", filename=None, clear=True):
which is great as name specifies the log file ...
I forgot what filename actually means is ist the specification?
filename tries to load the yaml file, so if the specification means the yaml file, then yes.
has this been completed?
@laszewsk we cannot go further because the .format method in labelmaker does not allow colons. https://stackoverflow.com/questions/19913504/how-do-i-escape-a-colon-in-python-format-when-using-kwargs
so, we cannot have labels with colons in them.. like if we wanted to show the time, like 10:55 AM.. it is not possible. https://github.com/cloudmesh/cloudmesh-cc/blob/0df34fa3bf205cb36aae8d0606874421c38dbb8b/cloudmesh/cc/labelmaker.py#L226
The logic for time reporting in the workflow needs to change
obviouslye we need to have
we also need
There is the possibility to just ad a dt to the workflow node just as we have created and modified, This way it is easy to freeze it as we will not change it after the progress is 100
I think this is easy to cahnge as we can just pass all values of a node into labelmaker and than through labels bick the right values to replace
We could ven add a flag such as relative=True and all times are reported relative to t0
THis is not completed and possibly now not relevant as we also need created and modified
we need a dt in labelmaker, we can se cloudmesh variables to record the time of a start of a workflow such as
dt.name where name is the name of the workflow
so dt may look like
{dt.name.%m/%d/%Y, %H:%M:%S}
where name is the name of the workflow
t0 is then used in variables as
t0_name
dt then parses the name and format and can tahn use
t0 = variables[f"t0_{name}"]