code":"278","message":"Quote mismatch: <End-of-Input>" using search_subset_in_native_views

[jnaff-coursera]

Describe what did you try to do with TM1py I'm wondering if having an ampersand (&) in the subset name is causing the issue. I called this: tm1dev.views.search_subset_in_native_views(dimension_name=dim, subset_name=subset) where dim = Department and subset = P&L Dept

Describe what's not working the way you expect Didn't get the expected result? Describe:

1. The result you expected. to get a list of public and private views if any contained the subset
2. A clear and concise description of what you are trying to do. to identify views if the subset assigned to them Version
   • TM1py 1.10.2
   • TM1 Server Version: unsure exact version, IBM Cloud customer

Additional context If you encounter an error, please add the error message and the stack trace Traceback (most recent call last): File "C:\Cubewise\TM1py\tm1py-samples-master\Administration\TM1GetSubsetsUsedInViews.py", line 27, in viewsPublic_And_Private = tm1dev.views.search_subset_in_native_views(dimension_name=dim, subset_name=subset) File "C:\Cubewise\TM1py\tm1py-samples-master\venv\lib\site-packages\TM1py\Services\ViewService.py", line 262, in search_subset_in_native_views response = self._rest.GET(url, **kwargs) File "C:\Cubewise\TM1py\tm1py-samples-master\venv\lib\site-packages\TM1py\Services\RestService.py", line 86, in wrapper self.verify_response(response=response) File "C:\Cubewise\TM1py\tm1py-samples-master\venv\lib\site-packages\TM1py\Services\RestService.py", line 490, in verify_response raise TM1pyRestException(response.text, TM1py.Exceptions.Exceptions.TM1pyRestException: Text: '{"error":{"code":"278","message":"Quote mismatch: "}}' - Status Code: 400 - Reason: 'Bad Request' - Headers: {'Content-Length': '84', 'Connection': 'keep-alive', 'Content-Encoding': 'gzip', 'Cache-Control': 'no-cache', 'Content-Type': 'application/json; charset=utf-8', 'OData-Version': '4.0'}

[jnaff-coursera]

I noticed this recent IBM APAR that might be related: https://www.ibm.com/support/pages/apar/PH50927

[gbryant-dev]

Hi @jnaff-coursera, having & in the object name would cause an issue as it’s used for url parameters. Try replacing it with %26 as a workaround. @MariusWirtz I think TM1py should handle this in build_url_friendly_object_name?

[jnaff-coursera]

Thank you. I tried that: subset = subset.replace('&', '%26'), but no views were returned. I created both a public and private view with P&L Department as a public and a private subset but search_subset_in_native_views returns empty lists.

[gbryant-dev]

Do the views get returned if you search for other subsets without& in the name for those same views? Are you able to share your script?

[MariusWirtz]

Hi @jnaff-coursera,

thanks for reporting the issue. I can reproduce it and I think I was able to fix it. Please upgrade to this hotfix branch of TM1py and let us know if it resolves the issue.

pip uninstall tm1py
pip install https://github.com/cubewise-code/tm1py/archive/refs/heads/issue/fix-ampersand-encoding-in-url.zip

[MariusWirtz]

Hi @jnaff-coursera, having & in the object name would cause an issue as it’s used for url parameters. Try replacing it with %26 as a workaround. @MariusWirtz I think TM1py should handle this in build_url_friendly_object_name?

@gbryant-dev you are right. TM1py didn't manage the encoding right.

[jnaff-coursera]

Thank you. I ran the 2 pip commands but I'm seeing the same error

[jnaff-coursera]

Here's the code:

with TM1Service(**config[source]) as tm1dev:
    dim = input("enter dimension name")
    subsets = tm1dev.dimensions.subsets.get_all_names(dim)
    for subset in subsets:
        print(dim + ":" + subset)
        error happens here: viewsPublic_And_Private = tm1dev.views.search_subset_in_native_views(dimension_name=dim, subset_name=subset)
        for viewsPublic_Or_Private in viewsPublic_And_Private:
            for view in viewsPublic_Or_Private:
                print("found " + dim + ":" + subset + ":" + view.cube + ":" + view.name)

Here's the error

    Traceback (most recent call last):
  File "C:\Cubewise\TM1py\tm1py-samples-master\Administration\TM1GetSubsetsUsedInViews.py", line 27, in <module>
    **viewsPublic_And_Private = tm1dev.views.search_subset_in_native_views(dimension_name=dim, subset_name=subset)**
  File "C:\Cubewise\TM1py\tm1py-samples-master\venv\lib\site-packages\TM1py\Services\ViewService.py", line 262, in search_subset_in_native_views
    response = self._rest.GET(url, **kwargs)
  File "C:\Cubewise\TM1py\tm1py-samples-master\venv\lib\site-packages\TM1py\Services\RestService.py", line 86, in wrapper
    self.verify_response(response=response)
  File "C:\Cubewise\TM1py\tm1py-samples-master\venv\lib\site-packages\TM1py\Services\RestService.py", line 490, in verify_response
    raise TM1pyRestException(response.text,
TM1py.Exceptions.Exceptions.TM1pyRestException: Text: '{"error":{"code":"278","message":"Quote mismatch: <End-of-Input>"}}' - Status Code: 400 - Reason: 'Bad Request' - Headers: {'Content-Length': '84', 'Connection': 'keep-alive', 'Content-Encoding': 'gzip', 'Cache-Control': 'no-cache', 'Content-Type': 'application/json; charset=utf-8', 'OData-Version': '4.0'}

[MariusWirtz]

Don't know if it's true, but here is my hypothesis: The fix doesn't apply because you are running Python in a virtual environment. So you are effectively still using the old version of TM1py.

If you use PyCharm please change the interpreter to System Interpreter. You can do it here: File -> Settings -> Project: <project-name> -> Python Interpreter

[MariusWirtz]

For this interpreter, TM1py isn't installed at all. Should be a quick fix.

In the interpreter settings of the PyCharm project, check where Python is installed. Probably it's something like this: "C:/Python39" or C:/program files/Python/Python310" or something along those lines.

Once we know, navigate to the "Python3/scripts" folder with the command prompt. This is where you find the pip package installer that we need to use: cd C:\Python310\Scripts

In this folder, you can re-run the pip command to install the TM1py hotfix branch. pip install https://github.com/cubewise-code/tm1py/archive/refs/heads/issue/fix-ampersand-encoding-in-url.zip

[jnaff-coursera]

Than you! I can now return the public view. I am not returning the private view yet. P&L Department is also saved as a private subset in a private view, this view is not getting returned but it might be my code.

[MariusWirtz]

Is TM1py running with the same user as the user that saved the view? If not, check out impersonation with TM1py :)

[jnaff-coursera]

Right. I did not consider that the private views would not return for other users. For some reason I thought TM1Py would return all views including private ones. Makes sense now. I am familiar with Arc's impersonation, however I do not see any documentation when I search "impersonation" for release 1.10.2

[jnaff-coursera]

I see how to now: tm1 = TM1Service(address="", port=12297, ssl=True, user="admin", password="", impersonate="Joe")

[gbryant-dev]

Than you! I can now return the public view. I am not returning the private view yet. P&L Department is also saved as a private subset in a private view, this view is not getting returned but it might be my code.

I think the problem lies in this line viewsPublic_And_Private = tm1dev.views.search_subset_in_native_views(dimension_name=dim, subset_name=subset)

search_subset_in_native_views returns two lists so you need to extract them like so:

private_views, public_views = tm1dev.views.search_subset_in_native_views (dimension_name=dim, subset_name=subset)

If you would like to loop through both lists in one go you can join them together using views = private_views + public_views

[jnaff-coursera]

Thank you @gbryant-dev !

[jnaff-coursera]

I am testing to retrieve a private view that I saved with my regular user account. I am an Admin. I am connecting TM1py using a cloud automation account that is an Admin user, however when I try to impersonate the Admin user, I see this error: Traceback (most recent call last): File "C:\Cubewise\TM1py\tm1py-samples-master\Administration\TM1GetSubsetsUsedInViews.py", line 25, in with TM1Service( File "C:\Python\Python310\lib\site-packages\TM1py\Services\TM1Service.py", line 16, in init self._tm1_rest = RestService(**kwargs) File "C:\Python\Python310\lib\site-packages\TM1py\Services\RestService.py", line 229, in init self.connect() File "C:\Python\Python310\lib\site-packages\TM1py\Services\RestService.py", line 249, in connect self._start_session( File "C:\Python\Python310\lib\site-packages\TM1py\Services\RestService.py", line 423, in _start_session response = self.GET(url=url, headers=additional_headers) File "C:\Python\Python310\lib\site-packages\TM1py\Services\RestService.py", line 73, in wrapper self.verify_response(response=response) File "C:\Python\Python310\lib\site-packages\TM1py\Services\RestService.py", line 517, in verify_response raise TM1pyRestException(response.text, TM1py.Exceptions.Exceptions.TM1pyRestException: Text: 'Error: Forbidden You have insufficient rights to impersonate the user specified! ' - Status Code: 403 - Reason: 'Forbidden' - Headers: {'Date': 'Sat, 08 Apr 2023 23:37:40 GMT', 'Server': 'Apache', 'Strict-Transport-Security': 'max-age=31536000; includeSubdomains', 'Content-Type': 'text/plain', 'Content-Length': '92', 'Content-Encoding': 'gzip', 'Set-Cookie': 'TM1SessionId=GpXnhYJHxn8S6KOVEjnaVn5aVEQ; Path=/tm1/api/tm1/; HttpOnly; Secure', 'X-Content-Type-Options': 'nosniff', 'Keep-Alive': 'timeout=5, max=100', 'Connection': 'Keep-Alive'}

mycompany_tm1_automation is an Admin user.

This code works, user ksmith is not an Admin with TM1Service( namespace="LDAP", user="mycompany_tm1_automation", password="mypassword", ssl=True, verify=True, async_requests_mode=True, base_url="https://mycompany.planning-analytics.ibmcloud.com/tm1/api/tm1/", impersonate='CAMID("pans:u:ksmith@mycompany.org")') as tm1dev

This does not work, jnaff is an Admin
with TM1Service( namespace="LDAP", user="mycompany_tm1_automation", password="mypassword", ssl=True, verify=True, async_requests_mode=True, base_url="https://mycompany.planning-analytics.ibmcloud.com/tm1/api/tm1/", impersonate='CAMID("pans:u:jnaff@mycompany.org")') as tm1dev

[jnaff-coursera]

Looks like this might be expected behavior. Admin users cannot impersonate other Admin users, correct?

[MariusWirtz]

Looks like this might be expected behavior. Admin users cannot impersonate other Admin users, correct?

Correct. This is something we might want to raise with IBM. Impersonation of admin users would generally be helpful.

[jnaff-coursera]

Thank you @MariusWirtz and @gbryant-dev for your quick responses.

[jnaff-coursera]

I am seeing memory spikes. I was running the TM1py script that loops through users and impersonates them to get their views. It was running for 90 minutes until I canceled it due to the memory threshold alert I received. When I looked at StatsForServer it was at 62GB for Total Memory Used, but the StatsByCube was only totaling 12GB so I'm wondering what was consuming the other 50GB. Any thoughts? Thank you!

[MariusWirtz]

I am seeing memory spikes. I was running the TM1py script that loops through users and impersonates them to get their views. It was running for 90 minutes until I canceled it due to the memory threshold alert I received. When I looked at StatsForServer it was at 62GB for Total Memory Used, but the StatsByCube was only totaling 12GB so I'm wondering what was consuming the other 50GB. Any thoughts? Thank you!

I created a new issue for the memory spikes. Please create one dedicated issue per topic.