Add a from_pro function to the Process class in TM1py

[wimgielis]

Describe what did you try to do with TM1py It would be nice to have hot promotion for TI processes, where the source is a, existing PRO file. Say, a process was created in development, and we want to promote it to production without bouncing the TM1 server.

Describe what's not working the way you expect This would be an enhancement.

Refer to: https://github.com/cubewise-code/tm1py-samples/issues/77

[scrambldchannel]

This is an interesting use case @wimgielis. As Marius already alluded to, the tricky part is parsing a .pro file to pull out the necessary fields to create an instance of the Process object. Once you have that, it can be created on the server.

I'm not sure I'm brave enough to try right now, it seems a bit hairy :) I wonder though if there's anything out there documenting the structure of the .pro files? Maybe someone involved in the Bedrock project has published something?

[wimgielis]

Hi Alexander,

You can start by opening a PRO file in Notepad++ for a process you also open in for example Architect Turbo Integrator. Then change a setting in the process, save and see what changes in the text file.

Here is a list of constants: file:///C:/ibm/cognos/tm1_64/TM1JavaApiDocs/constant-values.html If you look for (Ctrl-F) 573 for instance, you will find that it's about the code for the Advanced > Metadata tab.

Another topic of mine related to creating processes: https://www.tm1forum.com/viewtopic.php?f=3&t=14990&p=74343

I have Excel VBA code but it uses the old legacy VB API for TM1.

So, you have another meat on the plate for the coming weekend ? :-)

Best regards,

Wim

Op vr 2 okt. 2020 om 16:01 schreef Alexander Sutcliffe < notifications@github.com>:

This is an interesting use case @wimgielis https://github.com/wimgielis. As Marius already alluded to, the tricky part is parsing a .pro file to pull out the necessary fields to create an instance of the Process object. Once you have that, it can be created on the server.

I'm not sure I'm brave enough to try right now, it seems a bit hairy :) I wonder though if there's anything out there document the structure of the .pro files? Maybe someone involved in the Bedrock project has published something?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cubewise-code/tm1py/issues/383#issuecomment-702751379, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEDHULPFI2NDEOKRBWX2XOLSIXMLRANCNFSM4SBQZX3Q .

[scrambldchannel]

Thanks, that's interesting... I am looking at the IBM docs with trepidation ;)

I think this weekend I'd like to get outside a bit more! Let's see, those codes are really useful but it's still a bit fiddly

Cheers Alex

[adscheevel]

Does the source have to be a PRO file? I use TM1Py to hot promote from dev environment to PROD but I do that by connecting to both environments, getting the desired process from the source environment, and then create in the target environment with the process object I just got from Dev. Below is example script of what I've used in the past where I can change the name of the process when promoting.

tm1Dev = TM1Service(....)
tm1Prod = TM1Service(...)

SrcProcName = 'zTEMP - Hot Promote ~20201002'
TarProcName = 'zTEMP - Hot PromotED ~20201002'

oProcess = tm1Dev.processes.get(SrcProcName)
oProcess.name = TarProcName

tm1Prod.processes.create(oProcess)

[wimgielis]

Hello @adscheevel

I think there are 2 things to mention here:

• we could use a PRO file, since that would make most sense. We are not using text files (other than PRO files) or databases to store coding, variables, etc. So PRO files make sense, more than any other source, I would argue. It could be a download of an earlier TM1 model, or a model from a different customer, anything.

• I was thinking about a PRO file that is not an active process in a TM1 model. If we talk about promoting a process (as part of the }Processes dimension) from 1 model to a different model, we could use your approach. However, the idea was to make a PRO file available in a target TM1 model, where the source is a PRO text file (correct process syntax needed) and just sitting on the hard disk somewhere. For example, there could be a new Bedrock process, we download it, and promote it to a TM1 model without restarting TM1. Or I could have an own library of TI processes, I don't need all of the processes, but when I create or update a processs I would be happy to add it to certain TM1 models without having to restart the models.

Maybe...using TM1py to create a simple small TM1 model with just that 1 or a few TI processes (no cubes, dims, ...), then hot promoting like you said, would be a solution ? That's food for thought for @MariusWirtz probably :-)

[scrambldchannel]

Hi @wimgielis

@adscheevel's method sounds a good solution and you could, as you say, potentially create a server that is just repository for all processes and promote them for there. Presumably, you're using one of the native clients or Arc to create the processes in the first place which is going to be easier than trying to edit the pro file directly.

It's nice to be able to save things as text though if you want to take advantage of version control. I toyed with exporting TI processes, dimensions and cubes to JSON using TM1py's built-in methods and then editing the JSON but it was a bit fiddly to try to edit objects like large dimensions.

The direction IBM seems to be going is to manage TM1 models in git like this repo from Hubert. Processes are defined in two files, one containing the metadata for the process and one containing the lines of TI code which makes it a bit nicer to edit them. I haven't tested this though.

Anyway, the weather was horrible here yesterday so I did manage to get a rough proof of concept going for your original request. It's a bit clunky but seems to work at least in some cases. It might give you something to work with, no promises though :) I wrote a bit more about it here noting a few issues with handling embedded quotes and commas.

Create a dictionary of all codes and their values:

# codes to treat differently
multiline_codes = ['560', '561', '572', '573', '574', '575', '577', '578', '579', '580', '581', '582', '566']
multiline_codes_with_key = ['590','637']

# location of pro file to load
file = "}bedrock.cube.rule.processfeeders.pro"

with open(file, encoding='utf-8-sig') as f:

    process_dict = {}
    in_multiline = False
    in_multiline_with_key = False
    code = ''

    for line in f:
        if in_multiline:
            process_dict[code].append(line.replace('"', '').rstrip())
            lines = lines - 1
            if lines == 0:
                in_multiline = False
        elif in_multiline_with_key:
            fields = line.split(',')
            process_dict[code].append(fields[1].replace('"', '').rstrip())
            lines = lines - 1
            if lines == 0:
                in_multiline_with_key = False
        else:
            fields = line.split(',')
            code = fields[0]
            if code in multiline_codes:
                lines = int(fields[1])
                if lines > 0:
                    in_multiline = True
                process_dict[code] = []
            elif code in multiline_codes_with_key:
                lines = int(fields[1])
                if lines > 0:
                    in_multiline_with_key = True
                process_dict[code] = []
            else:
                # hacky way to deal with commas in the set values, eg where ',' is set as one of the delimiter character
                process_dict[code] = ''.join(fields[1:]).replace('"', '').rstrip() # hacky way to deal with commas in the set values

Try to build an instance of the Process class:

import TM1py

my_new_process = TM1py.Objects.Process(
    name=process_dict['602'],
    has_security_access=(True if process_dict['1217'] == 'True' else False),
    ui_data=process_dict['576'],
    prolog_procedure="\n".join(process_dict['572']),
    metadata_procedure="\n".join(process_dict['573']),
    data_procedure="\n".join(process_dict['574']),
    epilog_procedure="\n".join(process_dict['575']),
    datasource_type='None',
    datasource_ascii_decimal_separator=process_dict['588'],
    datasource_ascii_delimiter_char=process_dict['567'],
    datasource_ascii_delimiter_type='Character', # doesn't seem to have a corresponding code
    datasource_ascii_header_records=process_dict['569'],
    datasource_ascii_quote_character=process_dict['568'],
    datasource_ascii_thousand_separator=process_dict['589'],
    datasource_data_source_name_for_client=process_dict['585'],
    datasource_data_source_name_for_server=process_dict['586'],
    datasource_password=process_dict['565'],
    datasource_user_name=process_dict['564'],
    datasource_query=process_dict['566'],
    datasource_uses_unicode=process_dict['559'],
    datasource_view=process_dict['570'],
    datasource_subset=process_dict['571']
)

# now add parameters and variables

for index, item in enumerate(process_dict['560']):

    if process_dict['561'][index] == "2":
        parameter_type = "String"
        value = process_dict['590'][index]
    else:
        parameter_type = "Numeric"
        if process_dict['590'][index] == "":
            value = 0
        else:
            value = float(process_dict['590'][index])

    my_new_process.add_parameter(
        name=item,
        prompt=process_dict['637'][index],
        value=value,
        parameter_type=parameter_type
    )

for index, item in enumerate(process_dict['577']):

    variable_type = "String" if process_dict['578'][index] == "2" else "Numeric"        

    my_new_process.add_variable(
        name=item,
        variable_type=variable_type
    )    

Connect to TM1 and add the process:

import configparser

# establish connection / how you do this is up to you
config = configparser.ConfigParser()
config.read('config.ini')

with TM1py.Services.TM1Service(**config['tm1srv01']) as tm1:

    if tm1.processes.exists(my_new_process.name):
        tm1.processes.delete(my_new_process.name)

    response = tm1.processes.create(my_new_process)

    # check status of response
    print(response.status_code)

[wimgielis]

Many thanks Alexander !

I will review the TI codes and see if I can contribute here and there. Great job already 👍

Op zo 4 okt. 2020 om 16:07 schreef Alexander Sutcliffe < notifications@github.com>

Hi @wimgielis https://github.com/wimgielis

@adscheevel https://github.com/adscheevel's method sounds a good solution and you could, as you say, potentially create a server that is just repository for all processes and promote them for there. Presumably, you're using one of the native clients or Arc to create the processes in the first place which is going to be easier than trying to edit the pro file directly.

It's nice to be able to save things as text though if you want to take advantage of version control. I toyed with exporting TI processes, dimensions and cubes to JSON using TM1py's built-in methods and then editing the JSON but it was a bit fiddly to try to edit objects like large dimensions.

The direction IBM seems to be going is to manage TM1 models in git like this repo from Hubert https://github.com/Hubert-Heijkers/tm1-model-pony-music-backup. Processes are defined in two files, one containing the metadata for the process and one containing the lines of TI code which makes it a bit nicer to edit them. I haven't tested this though.

Anyway, the weather was horrible here yesterday so I did manage to get a rough proof of concept going for your original request. It's a bit clunky but seems to work at least in some cases. It might give you something to work with, no promises though :)

Create a dictionary of all codes and their values:

codes to treat differently

multiline_codes = ['560', '561', '572', '573', '574', '575', '577', '578', '579', '580', '581', '582', '566']

multiline_codes_with_key = ['590','637']

location of pro file to load

file = "}bedrock.cube.rule.processfeeders.pro"

with open(file, encoding='utf-8-sig') as f:

process_dict = {}

in_multiline = False

in_multiline_with_key = False

code = ''

for line in f:

    if in_multiline:

        process_dict[code].append(line.replace('"', '').rstrip())

        lines = lines - 1

        if lines == 0:

            in_multiline = False

    elif in_multiline_with_key:

        fields = line.split(',')

        process_dict[code].append(fields[1].replace('"', '').rstrip())

        lines = lines - 1

        if lines == 0:

            in_multiline_with_key = False

    else:

        fields = line.split(',')

        code = fields[0]

        if code in multiline_codes:

            lines = int(fields[1])

            if lines > 0:

                in_multiline = True

            process_dict[code] = []

        elif code in multiline_codes_with_key:

            lines = int(fields[1])

            if lines > 0:

                in_multiline_with_key = True

            process_dict[code] = []

        else:

            # hacky way to deal with commas in the set values, eg where ',' is set as one of the delimiter character

            process_dict[code] = ''.join(fields[1:]).replace('"', '').rstrip() # hacky way to deal with commas in the set values

Try to build an instance of the Process class:

import TM1py

my_new_process = TM1py.Objects.Process(

name=process_dict['602'],

has_security_access=(True if process_dict['1217'] == 'True' else False),

ui_data=process_dict['576'],

prolog_procedure="\n".join(process_dict['572']),

metadata_procedure="\n".join(process_dict['573']),

data_procedure="\n".join(process_dict['574']),

epilog_procedure="\n".join(process_dict['575']),

datasource_type='None',

datasource_ascii_decimal_separator=process_dict['588'],

datasource_ascii_delimiter_char=process_dict['567'],

datasource_ascii_delimiter_type='Character', # doesn't seem to have a corresponding code

datasource_ascii_header_records=process_dict['569'],

datasource_ascii_quote_character=process_dict['568'],

datasource_ascii_thousand_separator=process_dict['589'],

datasource_data_source_name_for_client=process_dict['585'],

datasource_data_source_name_for_server=process_dict['586'],

datasource_password=process_dict['565'],

datasource_user_name=process_dict['564'],

datasource_query=process_dict['566'],

datasource_uses_unicode=process_dict['559'],

datasource_view=process_dict['570'],

datasource_subset=process_dict['571']

)

now add parameters and variables

for index, item in enumerate(process_dict['560']):

if process_dict['561'][index] == "2":

    parameter_type = "String"

    value = process_dict['590'][index]

else:

    parameter_type = "Numeric"

    if process_dict['590'][index] == "":

        value = 0

    else:

        value = float(process_dict['590'][index])

my_new_process.add_parameter(

    name=item,

    prompt=process_dict['637'][index],

    value=value,

    parameter_type=parameter_type

)

for index, item in enumerate(process_dict['577']):

variable_type = "String" if process_dict['578'][index] == "2" else "Numeric"

my_new_process.add_variable(

    name=item,

    variable_type=variable_type

)

Connect to TM1 and add the process:

import configparser

establish connection / how you do this is up to you

config = configparser.ConfigParser()

config.read('config.ini')

with TM1py.Services.TM1Service(**config['tm1srv01']) as tm1:

if tm1.processes.exists(my_new_process.name):

    tm1.processes.delete(my_new_process.name)

response = tm1.processes.create(my_new_process)

# check status of response

print(response.status_code)

I wrote a bit more about it here https://scrambldchannel.github.io/hot-promotion-of-tm1-pro-file.html#hot-promotion-of-tm1-pro-file

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/cubewise-code/tm1py/issues/383#issuecomment-703260960, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEDHULOAWK7S3WHWGHDD67LSJB6SHANCNFSM4SBQZX3Q .

--

---

Best regards / Beste groeten,

Wim Gielis MS Excel MVP 2011-2014 https://www.wimgielis.com http://www.wimgielis.be

[wimgielis]

Hi Alexander,

Is it true that you explicit set None data source for the process ? If yes, maybe you could have a look at property 562, which is any of these:

• ODBC
• CHARACTERDELIMITED
• POSITIONDELIMITED
• ODBOCUBE
• ODBODIMENSION
• ODBOMDXQUERY
• VIEW
• SUBSET
• COGNOSDIMENSION
• COGNOSPACKAGE
• COGNOSQUERY
• NULL

---

Best regards / Beste groeten,

Wim Gielis MS Excel MVP 2011-2014 https://www.wimgielis.com http://www.wimgielis.be

Op zo 4 okt. 2020 om 20:38 schreef Wim Gielis notifications@github.com:

Many thanks Alexander !

I will review the TI codes and see if I can contribute here and there. Great job already 👍

Op zo 4 okt. 2020 om 16:07 schreef Alexander Sutcliffe < notifications@github.com>

Hi @wimgielis https://github.com/wimgielis

@adscheevel https://github.com/adscheevel's method sounds a good solution and you could, as you say, potentially create a server that is just repository for all processes and promote them for there. Presumably, you're using one of the native clients or Arc to create the processes in the first place which is going to be easier than trying to edit the pro file directly.

It's nice to be able to save things as text though if you want to take advantage of version control. I toyed with exporting TI processes, dimensions and cubes to JSON using TM1py's built-in methods and then editing the JSON but it was a bit fiddly to try to edit objects like large dimensions.

The direction IBM seems to be going is to manage TM1 models in git like this repo from Hubert https://github.com/Hubert-Heijkers/tm1-model-pony-music-backup. Processes are defined in two files, one containing the metadata for the process and one containing the lines of TI code which makes it a bit nicer to edit them. I haven't tested this though.

Anyway, the weather was horrible here yesterday so I did manage to get a rough proof of concept going for your original request. It's a bit clunky but seems to work at least in some cases. It might give you something to work with, no promises though :)

Create a dictionary of all codes and their values:

codes to treat differently

multiline_codes = ['560', '561', '572', '573', '574', '575', '577', '578', '579', '580', '581', '582', '566']

multiline_codes_with_key = ['590','637']

location of pro file to load

file = "}bedrock.cube.rule.processfeeders.pro"

with open(file, encoding='utf-8-sig') as f:

process_dict = {}

in_multiline = False

in_multiline_with_key = False

code = ''

for line in f:

if in_multiline:

process_dict[code].append(line.replace('"', '').rstrip())

lines = lines - 1

if lines == 0:

in_multiline = False

elif in_multiline_with_key:

fields = line.split(',')

process_dict[code].append(fields[1].replace('"', '').rstrip())

lines = lines - 1

if lines == 0:

in_multiline_with_key = False

else:

fields = line.split(',')

code = fields[0]

if code in multiline_codes:

lines = int(fields[1])

if lines > 0:

in_multiline = True

process_dict[code] = []

elif code in multiline_codes_with_key:

lines = int(fields[1])

if lines > 0:

in_multiline_with_key = True

process_dict[code] = []

else:

hacky way to deal with commas in the set values, eg where ',' is set

as one of the delimiter character

process_dict[code] = ''.join(fields[1:]).replace('"', '').rstrip() # hacky way to deal with commas in the set values

Try to build an instance of the Process class:

import TM1py

my_new_process = TM1py.Objects.Process(

name=process_dict['602'],

has_security_access=(True if process_dict['1217'] == 'True' else False),

ui_data=process_dict['576'],

prolog_procedure="\n".join(process_dict['572']),

metadata_procedure="\n".join(process_dict['573']),

data_procedure="\n".join(process_dict['574']),

epilog_procedure="\n".join(process_dict['575']),

datasource_type='None',

datasource_ascii_decimal_separator=process_dict['588'],

datasource_ascii_delimiter_char=process_dict['567'],

datasource_ascii_delimiter_type='Character', # doesn't seem to have a corresponding code

datasource_ascii_header_records=process_dict['569'],

datasource_ascii_quote_character=process_dict['568'],

datasource_ascii_thousand_separator=process_dict['589'],

datasource_data_source_name_for_client=process_dict['585'],

datasource_data_source_name_for_server=process_dict['586'],

datasource_password=process_dict['565'],

datasource_user_name=process_dict['564'],

datasource_query=process_dict['566'],

datasource_uses_unicode=process_dict['559'],

datasource_view=process_dict['570'],

datasource_subset=process_dict['571']

)

now add parameters and variables

for index, item in enumerate(process_dict['560']):

if process_dict['561'][index] == "2":

parameter_type = "String"

value = process_dict['590'][index]

else:

parameter_type = "Numeric"

if process_dict['590'][index] == "":

value = 0

else:

value = float(process_dict['590'][index])

my_new_process.add_parameter(

name=item,

prompt=process_dict['637'][index],

value=value,

parameter_type=parameter_type

)

for index, item in enumerate(process_dict['577']):

variable_type = "String" if process_dict['578'][index] == "2" else "Numeric"

my_new_process.add_variable(

name=item,

variable_type=variable_type

)

Connect to TM1 and add the process:

import configparser

establish connection / how you do this is up to you

config = configparser.ConfigParser()

config.read('config.ini')

with TM1py.Services.TM1Service(**config['tm1srv01']) as tm1:

if tm1.processes.exists(my_new_process.name):

tm1.processes.delete(my_new_process.name)

response = tm1.processes.create(my_new_process)

check status of response

print(response.status_code)

I wrote a bit more about it here < https://scrambldchannel.github.io/hot-promotion-of-tm1-pro-file.html#hot-promotion-of-tm1-pro-file

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub < https://github.com/cubewise-code/tm1py/issues/383#issuecomment-703260960>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AEDHULOAWK7S3WHWGHDD67LSJB6SHANCNFSM4SBQZX3Q

.

--

---

Best regards / Beste groeten,

Wim Gielis MS Excel MVP 2011-2014 https://www.wimgielis.com http://www.wimgielis.be

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/cubewise-code/tm1py/issues/383#issuecomment-703297401, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEDHULJVYR7D6FETTXPVEB3SJC6KZANCNFSM4SBQZX3Q .

[scrambldchannel]

Hi Alexander, Is it true that you explicit set None data source for the process ?

Ah, well spotted, I must have hacked that in. I'll take a look once I've had some sleep :)

[cubewise-tryan]

Hi all,

Just a warning on going down this path, it is easy enough for simple examples but it will take a lot of effort to make it work generically across all of the options that are available in TI. There are lots of little quirks in the pro file format, we know from our experience parsing it for Pulse.

It is much easier to just extract the process via the REST API and then updating it using that source. It isn't very difficult to start up a TM1 server from the command line.

[MariusWirtz]

I tend to agree with @cubewise-tryan on this one.

Happy to accept a PR though if someone can actually make it work :)