hpc-io / pdc

Proactive Data Containers (PDC) software provides an object-centric API and a runtime system with a set of data object management services. These services allow placing data in the memory and storage hierarchy, performing data movement asynchronously, and providing scalable metadata operations to find data objects.
https://pdc.readthedocs.io
Other
13 stars 12 forks source link

An API to get an object or to open an object using the metadata ID of an object #207

Open zhangwei217245 opened 3 months ago

zhangwei217245 commented 3 months ago

What does this feature solve or improve?

Currently, the obj_id at client side is only unique within one client process, and is not globally unique. The only globally unique numerical ID for object is the meta_id at the server side.

Also, the client side takes object_name + timestep to be unique identifier of objects, but it is taking too many bytes than a single numerical ID.

It would be good to create an API for retrieving object or open object using the actual metadata ID of that object, and we should consider treating the metadata ID of an object to the ultimate identifier for objects in PDC.

Describe the solution you'd like

These APIs are needed :

  1. pdc_metadata_t * PDC_obj_retrieve_metadata(uint64_t meta_id)
  2. perr_t PDC_obj_open_by_meta_id(uint64_t meta_id)

Describe alternatives you've considered

Also, a bigger refactoring of PDC would be a re-designing of the ID of objects in PDC, and also considering elimination of client-side mapping of objects.... We can do client-side caching of the objects, but doesn't necessarily have to perform client-side object mapping.

We should unify the ID of any PDC object to be a numerical ID that is globally unique. The current ID mechanism is already used everywhere in PDC and it can take a lot of effort to change the ID system. But, we can keep this as a long term effort and we don't have to immediately change the design of object IDs. Once we get chance and time, we should do it the right way.

Additional Information

N/A