Issue with convert View using function NativeView.from_json

[ZHENGHECHU]

Describe what did you try to do with TM1py Hello, i run the following script aiming to convert Views between serveurs but encounters problems using function NativeView.from_json

Script:

#use 'get' function to get view and the store the view in a txt.file 
with TM1Service(**config["Test1"]) as tm1:
    view_get = tm1.views.get(cube_name, view_name)
    view_get = json.dumps(json.loads(view_get.body), indent=4)
    file_path_view = os.path.join(script_dir, folder, "{}.txt".format(view_name))
    with open(file_path_view, "w", encoding="utf-8") as file:
        file.write(view_get)

#read the same txt.file and use 'NativeView.from_json' function (here take NativeView for example) to update_or_create view in target instance. 
with TM1Service(**config["Test2"]) as tm1_target:
   file_path_view = os.path.join(
                    script_dir, folder, "{}.txt".format(view_name)
                )
    with open(file_path_view, "r", encoding="utf-8") as file:
        view_put = file.read()
        view_put = NativeView.from_json(view_put)
        tm1_target.views.update_or_create(view_put)

Furthur information why i dont convert views between serveurs directly, beacuase i want the target instance only read the txt.file to import the View

The output in txt.file:

{
    "@odata.type": "ibm.tm1.api.v1.NativeView",
    "Name": "Default",
    "Columns": [
        {
            "Subset": {
                "Hierarchy@odata.bind": "Dimensions('Date')/Hierarchies('Date')",
                "Elements@odata.bind": [
                    "Dimensions('Date')/Hierarchies('Date')/Elements('90_days')"
                ]
            }
        },
        {
            "Subset": {
                "Hierarchy@odata.bind": "Dimensions('Measure')/Hierarchies('Measure')",
                "Expression": "TM1SubsetAll([Measure])"
            }
        }
    ],
    "Rows": [
        {
            "Subset": {
                "Hierarchy@odata.bind": "Dimensions('Heure')/Hierarchies('Heure')",
                "Expression": "{DRILLUPMEMBER({[Heure].[Heure].Members}, {[Heure].[Heure].[Total heure]})}"
            }
        }
    ],
    "Titles": [
        {
            "Subset": {
                "Hierarchy@odata.bind": "Dimensions('Location')/Hierarchies('Location')",
                "Expression": "{[Location].[Location].Members}"
            },
            "Selected@odata.bind": "Dimensions('Location')/Hierarchies('Location')/Elements('I1')"
        },
        {
            "Subset": {
                "Hierarchy@odata.bind": "Dimensions('Scope')/Hierarchies('Scope')",
                "Expression": "{[Scope].[Scope].Members}"
            },
            "Selected@odata.bind": "Dimensions('Scope')/Hierarchies('Scope')/Elements('0')"
        }
    ],
    "SuppressEmptyColumns": false,
    "SuppressEmptyRows": false,
    "FormatString": "0.#########"
}

ERROR MESSAGE:

    view_put = NativeView.from_json(view_put)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python\Python311\Lib\site-packages\TM1py\Objects\NativeView.py", line 225, in from_json
    return NativeView.from_dict(view_as_dict, cube_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python\Python311\Lib\site-packages\TM1py\Objects\NativeView.py", line 232, in from_dict
    if selection['Subset']['Name'] == '':
       ~~~~~~~~~~~~~~~~~~~^^^^^^^^
KeyError: 'Name'

[ZHENGHECHU]

TM1 version: 11.2 TM1py version: 1.11.3

[MariusWirtz]

Thanks for raising this issue @ZHENGHECHU I will look into this.

For the moment, as a workaround, I would like to propose two alternative approaches:

1. don't persist the view in a file and connect to both instances at the same time:

   with TM1Service(**config["Test1"]) as tm1:
   view = tm1.views.get(cube_name, view_name)
   
   with TM1Service(**config["Test2"]) as tm1_target:
       tm1_target.views.update_or_create(view)

2. pickle and unpickle the view object instead of writing and reading JSON.

   
   import pickle

with TM1Service(**config["Test1"]) as tm1: view = tm1.views.get(cube_name, view_name) with open("view", "wb") as file: pickle.dump(view, file)

with TM1Service(**config["Test2"]) as tm1_target:

with open("view", "rb") as file:
    view = pickle.load(file)    

tm1_target.views.update_or_create(view)

[ZHENGHECHU]

Hello @MariusWirtz, I appreciate your response and the valuable alternative suggestions you provided. However, I currently have a pressing need to store the view in a file while ensuring that the structure remains comprehensible. Your assistance in finding a suitable solution would be greatly appreciated. Kindly keep me informed by tagging me. thank you for your help in advance.

[MariusWirtz]

@ZHENGHECHU same as in #948 the problem is that TM1 returns a slightly different JSON when you read a chore than the JSON that must be posted when creating a chore.

I am working on a fix here: https://github.com/cubewise-code/tm1py/pull/955 I will add a few more test cases, but you can already upgrade TM1py to this feature branch and test if it resolves the issue.

pip uninstall tm1py
pip install https://github.com/cubewise-code/tm1py/archive/refs/heads/feature/make-native-view-from-json-robust.zip

You will need to make a small change to your code though. Make sure to pass the cube name to the from_json call.

view_put = NativeView.from_json(view_put, cube_name="Sales")

[ZHENGHECHU]

Hello @MariusWirtz,

Thank you for your reply,

Following the upgrade of TM1py to the feature branch, I continue to experience certain challenges. Views that include the 'Expression' element in their output exhibit no issues when utilizing the NativeView.from_json function, those lacking 'Expression' component are not functioning as intended.

Work well { "@odata.type": "ibm.tm1.api.v1.NativeView", "Name": "ALL", "Columns": [], "Rows": [ { "Subset": { "Hierarchy@odata.bind": "Dimensions('Geography')/Hierarchies('Geography')", "Expression": "TM1SubsetAll([Geography])" } }]},

didnt work

{ "@odata.type": "ibm.tm1.api.v1.NativeView", "Name": "Conso Prod 5144", "Columns": [ { "Subset": { "Hierarchy@odata.bind": "Dimensions('Collection')/Hierarchies('Collection')", "Elements@odata.bind": [ "Dimensions('Collection')/Hierarchies('Collection')/Elements('22K')" ] } }]},

Error message:

`Traceback (most recent call last): File "c:\TM1py_views.py", line 90, in view_put = NativeView.from_json(view_put, cube_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python\Python311\Lib\site-packages\TM1py\Objects\NativeView.py", line 226, in from_json return NativeView.from_dict(view_as_dict, cube_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python\Python311\Lib\site-packages\TM1py\Objects\NativeView.py", line 234, in from_dict subset = AnonymousSubset.from_dict(view_title['Subset']) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python\Python311\Lib\site-packages\TM1py\Objects\Subset.py", line 229, in from_dict if not subset_as_dict['Expression'] else None)

KeyError: 'Expression'`

[MariusWirtz]

Thanks for the catch @ZHENGHECHU

I updated the branch to handle this case. The bugfix is still WIP and missing unit tests, but feel free to upgrade and test already!

pip uninstall tm1py
pip install https://github.com/cubewise-code/tm1py/archive/refs/heads/feature/make-native-view-from-json-robust.zip

I will add test cases later this week.

[ZHENGHECHU]

Hello @MariusWirtz, Thank you for your dedication. Subsequent to the integration of the upgraded TM1py into the feature branch, I have observed that it functions effectively for the majority of views. However, a formatting concern has arisen. Specifically, when an element's name includes the '&' character, such as 'F & Sales', and this element is encompassed within views, the output in the generated text file will appear as 'F %26 Sales'. And the same situation, when the element contains '%', in the output file it become '%25'. Consequently, the functionality of the NativeView.from_json function becomes compromised.

While I surmise that this issue might be attributed to a reason I encountered in online sources, I cannot confirm its exact cause at this juncture.

when including '&' directly in the API request parameters without proper URL encoding, it can be misinterpreted by the server.

Here is an exmple of output { "@odata.type": "ibm.tm1.api.v1.NativeView", "Name": "test", "Rows": [ { "Subset": { "Hierarchy@odata.bind": "Dimensions('Collection')/Hierarchies('Collection')", "Elements@odata.bind": [ "Dimensions('Collection')/Hierarchies('Collection')/Elements('element1')", "Dimensions('Collection')/Hierarchies('Collection')/Elements('F %26 B Sales')"]}}]}

thanks for your help in advance

[MariusWirtz]

Thanks for the finding! Of course, we need to unescape the URL encoding in object names in the OData URLs.

I included this in the branch. Please upgrade and try again.

pip uninstall tm1py
pip install https://github.com/cubewise-code/tm1py/archive/refs/heads/feature/make-native-view-from-json-robust.zip

[MariusWirtz]

Resolved with #955