dotsdl commented 3 years ago

Description

This PR features a new PortalClient client interface, functioning as a MVP demonstrator for the following design features:

local client caching of COMPLETE procedure dataset records, with in-memory (transient) caching and filesystem (persistent) caching; designed around datasets being accessed far more frequently than they are added, and COMPLETE records not changing
use of __getitem__ for BaseProcedureDataset subclasses, such as OptimizationDataset as the access pattern of choice for getting records from compute specs
moving .data to ._data from top-level interface in Collection subclasses, treating it as an implementation detail and guiding users away from parsing it directly
movement of other low-level methods from visible interface to _ methods
use of progress bars via tqdm to clearly indicate server communication from an action, feedback to users on wait times
minimal use of pandas, in particular as a container for complex objects such as OptimizationRecords, where a dict or list will do; pandas is a great tool, but can make the client cumbersome to use with low value-add in many cases

The approach taken here is informed by user feedback collected in late 2020 on the difficulty of using the current FractalClient. We hope to make the use of the client more obvious for users by virtue of its exposed methods, interaction patterns, and responsiveness, while also improving performance for repeated access of the same dataset(s).

This approach is meant to inspire discussion, and we are early enough in the process to alter course. Please do offer feedback!

Changelog description

Status

[ ] Code base linted
[ ] Ready to go

codecov[bot] commented 3 years ago

Codecov Report

Merging #661 (5f00dd0) into master (a49f51d) will increase coverage by 0.01%. The diff coverage is 100.00%.

:exclamation: Current head 5f00dd0 differs from pull request most recent head 2a611ce. Consider uploading reports for the commit 2a611ce to get more accurate results