api.get_dataframe (and therefore DataFrame.from_deeporigin) can take extremely long because:
2 calls to list_database_rows and describe_row take place in series when they could take place in parallel (this is a problem of 700ms vs 300ms)
(more serious problem). there is a place where N DescriebFile calls are made in series, where N is the number of files in the database. this means that the time it takes to fetch data grows with the number of files linearly, and the time for each file resolution is ~350ms.
solution
[ ] parallelize 1
[ ] parallelize calls to DescribeFile
technical discussion and changes
[x] refactor code so that DescribeFile calls are isolated in a tight loop
[ ] write helper functions to create a configured async client
[ ] convert that tight loop into a parallel loop using async/await
[ ] use asyncio to bundle the 2 calls in problem 1 together
[ ] remove a semi-circular import where _api imports pandas and api
problem description
api.get_dataframe(and thereforeDataFrame.from_deeporigin) can take extremely long because:list_database_rowsanddescribe_rowtake place in series when they could take place in parallel (this is a problem of 700ms vs 300ms)DescriebFilecalls are made in series, where N is the number of files in the database. this means that the time it takes to fetch data grows with the number of files linearly, and the time for each file resolution is ~350ms.solution
technical discussion and changes
DescribeFilecalls are isolated in a tight loop