search

NCEAS / morpho

Morpho metadata editor
GNU General Public License v2.0
3 stars 1 forks source link

D1Object in Morpho still uses byte array as data source #970

Closed mbjones closed 6 years ago

mbjones commented 6 years ago

Author Name: Jing Tao (Jing Tao) Original Redmine Issue: 5796, https://projects.ecoinformatics.org/ecoinfo/issues/5796 Original Date: 2013-01-18 Original Assignee: Jing Tao

When morpho reads and saves the data package to the DataONE network, it handle the D1Objects. The entities, metadata and ore document will be transformed to the D1Object. However, the D1Object is still using the byt arrary as data source. That means we may put a big data file into the memory. This may cause some memory issue. The orignal design is that the D1Object will have a class attribute - a DataStore interface. The DataStore interface get and set methods. Those methods are handling InputStream and OutputStream, rather than byte array.

mbjones commented 6 years ago

Original Redmine Comment Author Name: ben leinfelder (ben leinfelder) Original Date: 2013-01-23T15:22:51Z

We discussed the following changes to D1Object. -add: getDataInputStream() and setDataInputStream() -modify: getData() and setData() to operate without actually storing the byte[] -use the stream-based methods in Morpho.

D1Object will likely defer to the D1Client (with cache enabled) to retrieve the data input stream initially.

mbjones commented 6 years ago

Original Redmine Comment Author Name: Jing Tao (Jing Tao) Original Date: 2013-01-26T01:25:02Z

I am thinking about the two methods we will add (in comment 1):

getDataInputStream() and setDataInputStream()

I think we should call getDataOutputStream()

So we should have a way to persistent the data in D1Object.

I believe the DataStore interface concept is still very useful. You may find it at; https://repository.dataone.org/documents/Projects/cicore/architecture/api-documentation/source/design/morpho/images/morpho-class-diagram.png

DataStore has two methods:

OutputStream get(Identifier id); void set(Identifier id, InputStream input);

mbjones commented 6 years ago

Original Redmine Comment Author Name: Jing Tao (Jing Tao) Original Date: 2013-01-28T18:36:57Z

create an issue on the dataone redmine. So the dataone persons can know what we will do. https://redmine.dataone.org/issues/3523

mbjones commented 6 years ago

Original Redmine Comment Author Name: Jing Tao (Jing Tao) Original Date: 2013-02-08T00:50:03Z

Now we use DataSource for D1Object to replace the byte[]

mbjones commented 6 years ago

Original Redmine Comment Author Name: Redmine Admin (Redmine Admin) Original Date: 2013-03-27T21:31:42Z

Original Bugzilla ID was 5796