MicroStrategy / mstrio-py

Python integration for MicroStrategy
Apache License 2.0
90 stars 60 forks source link

Unable to convert microstrategy report to dataframe. MicroStrategy API #55

Closed volom closed 3 years ago

volom commented 3 years ago

Can you help me please with exporting reports from Microstrategy with an API key using Python?

#import libraries
from mstrio import connection
from mstrio.application_objects.report import Report
import pandas as pd

# connection to project folder in Microstrategy
conn = connection.Connection(
    base_url="BASE_URL",
    username="USERNAME",
    password="PASSWORD",
    login_mode = 16,
    ssl_verify=False,
    project_name='PROJECT_NAME'
)

# connection to report
my_report = Report(connection=conn, report_id='REPORT_ID', parallel=True)

# applying filters to report

my_report.apply_filters(attributes=[x['id'] for x in list(my_report.attributes)],
                        attr_elements=attr_request). # variable "attr_request" consists my requested info

# converting to pandas DataFrame
df = my_report.to_dataframe()

The problem is in converting to pandas DataFrame. Despite the large initial report the filtered one has to be smaller and easy to convert. But in result "my_report.to_dataframe()" terminates the connection after a long unload.

Is there a way to reduce the initial report with applied filters?

I checked on smaller reports, the filter is applied and loaded normally. I think that Python loads the entire report into memory and applies the filter when converting to a dataframe. How to convert an already filtered report to a dataframe?

ignacyhologa commented 3 years ago

Hi volom, Could you clarify "terminates the connection after a long unload" The report object will only be populated with report data when to_dataframe() is called. Given that the filters have been applied properly it should download the subset of the report as you requested. Could you share the error you are experiencing?

volom commented 3 years ago

Hi volom, Could you clarify "terminates the connection after a long unload" The report object will only be populated with report data when to_dataframe() is called. Given that the filters have been applied properly it should download the subset of the report as you requested. Could you share the error you are experiencing?

Hi Ignacy Hołoga! Thank you very much for the responsiveness!

The problem is not the connection. It works well. The problem is that the report I'm connecting to is quite large, but if I filtered the report, it gets small (in python I need the filtered small dataset). my_report.to_dataframe() download the whole dataset from the server and then in python filters it (so the error «connection timeout» appears because downloading takes a long time), but I need to download the already filtered dataset from a server which takes less time so connection timeout error should not appear.

Is there any way to create a report with all filters in python and then send it to MicroStrategy or any other ways to solve this issue? Sorry for the possible confusion in the explanation. I am a newcomer in MicroStrategy:)

urszulajaczewska commented 3 years ago

Hi @volom, to answer your last question, try updating mstrio. It may work correctly now after some updates. Furthermore, you can try assigning None in attributes instead of list. If none of the above works, unfortunately it won't work because we send request for full content and filter it locally.

apiotrowskiMicroStrategy commented 3 years ago

Issue addressed and workaround found.