Improve Messagelog method

[scrumthing]

When I prepared my presentation for tm1 horizon next week I realized that the current response for the message log method is not directly usable it being more or less a list of lines. For further querying it you need to convert it to a different object (in my case a pandas dataframe) and translate the timestamp properly and get more options for filtering it. As I am not an expert for the message log I want to use the issue to start a discussion on that topic.

• What are your thoughts in that?
• What kind of useful information can you get from there? As I mentioned here I was thinking about feeder load time for example.
• How should the customization of the log (via logproperties file) be handled?

    def get_message_log_entries(self, reverse: bool = True, top: int = None, **kwargs) -> Dict:
        reverse = 'true' if reverse else 'false'
        url = '/api/v1/MessageLog(Reverse={})'.format(reverse)
        if top:
            url += '?$top={}'.format(top)
        response = self._rest.GET(url, **kwargs)
        return response.json()['value']```

[rkvinoth]

Hi @scrumthing

Just a small suggestion for the code blocks in your comments. Here is an example... enclose the multi-line code with ``` and add the language identifier (CASE INSENSITIVE) to enable syntax highlighting.

If you follow the above approach, your comment would look like:

def get_message_log_entries(self, reverse: bool = True, top: int = None, **kwargs) -> Dict: 
    reverse = 'true' if reverse else 'false' 
    url = '/api/v1/MessageLog(Reverse={})'.format(reverse) 
    if top: 
        url += '?$top={}'.format(top) 
    response = self._rest.GET(url, **kwargs) 
    return response.json()['value']

We have since and until arguments added in the latest version.. you should upgrade soon.

[scrumthing]

Changed the issue accordingly. Thanks for the hint!

We have since and until arguments added in the latest version.. you should upgrade soon.

Yes. It is on the agenda but after next week. Don't want to risk to break anything on my side. ;-)

But anyway I was talking about more than just since and until and more about the general approacht towards the message log. Do we just retrieve it and leave it to the user to parse it or do we want to cook some tm1 knowledge into the method(s) for that?

[rkvinoth]

Well the output of get_message_log_entries is a list of dictionaries and I think its better to stop TM1py's work there and let the developer use his own creativity.

if you have any difficulties turning the message log to a usable Dataframe, feel free to follow the below code.

from TM1py.Services import TM1Service
import pandas as pd

with TM1Service() as tm1:
    df = pd.DataFrame(tm1.server.get_message_log_entries(top=100))
    df.TimeStamp = pd.to_datetime(df.TimeStamp)

[scrumthing]

Thanks for the sample. It is way shorter than my method. :-) I had no problem converting the message log. My idea was to provide certain properties or filter options to get specific information faster from the message log. For example the startup time or the time it takes to process feeders for certain cubes. Or maybe just retrieve errors to reduce the size of the response.

In other cases tm1py does this as well. For example the method to retrieve the product version ( get_product_version ) is a specific use case of the get configuration method for faster results. :-)

[macsir]

For log analysis, I would use python to decode the log file directly on the server rather than using rest api as the log could be huge. Wrapping up api function to extract through network is not good idea and detailed log format could be very different.

[rkvinoth]

For example the startup time or the time it takes to process feeders for certain cubes. Or maybe just retrieve errors to reduce the size of the response.

We should be able to add those filters...

@macsir ... yup I would do the same. But it will be kind of messy unless you know your way around python file handling.

[MariusWirtz]

For log analysis, I would use python to decode the log file directly on the server rather than using rest api as the log could be huge. Wrapping up api function to extract through the network is not good idea and detailed log format could be very different.

Python does not necessarily run on the same machine as the TM1 server. You may not even be able to access the log folder when TM1 runs in the cloud.

I am not too concerned about the size of the message log generally. All TM1 models I worked with in the past had a reasonable log config that channeled limited information into the message log, to keep it human readable.

For example the startup time or the time it takes to process feeders for certain cubes. Or maybe just retrieve errors to reduce the size of the response.

We should be able to add those filters...

Yeah, I think so too. It would further reduce the risk of retrieving overly large chunks as @macsir mentioned. Ideally, we could filter by log level e.g. (INFO, ERROR, DEBUG)

[scrumthing]

I would like to filter by type as well, I. e. tm1.server, tm1.dimension, tm1.cube