search

isetbio / RemoteDataToolbox

Matlab utilities for managing reading and writing of data files stored on a remote web server.
6 stars 6 forks source link

Seems like a bug in retrieving artifacts #44

Closed wandell closed 8 years ago

wandell commented 8 years ago

The matlab code below is failing. Please help. Thanks ...

rdt = RdtClient('isetbio'); % Works OK rdt.crp('');

rdt =

repositoryName: isetbio repositoryUrl: http://52.32.77.154/repository/isetbio serverUrl: http://52.32.77.154/ workingRemotePath: Root username: guest password: **** verbosity: 0

artifacts = rdt.searchArtifacts('RedRose','type','mat'); % Works OK artifacts = 

  artifactId: 'RedRose'
 description: ''
   localPath: ''
        name: ''
  remotePath: 'resources/scenes/multiband/scien/2004'
repositoryId: 'isetbio'
        type: 'mat'
         url: [1x100 char]
     version: '1'

% But the read fails, see comment below data = rdt.readArtifact(artifacts.artifactId);

%%%% % Returns the error from gradle (Note the http: address is ....isetbio//RedRose/, note the '//' after isetbio) %%%

wandell commented 8 years ago

Sorry for all the messages ... The '//' bug above might have been unimportant. Not sure. Here is a similar failure, but without the '//' error. Sigh.

%% ieInit

%% Create the rdt object and open browser rd = RdtClient('isetbio'); rd =

repositoryName: isetbio repositoryUrl: http://52.32.77.154/repository/isetbio serverUrl: http://52.32.77.154/ workingRemotePath: resources/scene username: guest password: **** verbosity: 0

%% Our files are all version '1' at this point. fileVersion = '1';

% Here is an example remote directory. rd.crp('/resources/scene');

%% Problems here: % Currently only returns 30 elements % 'type' is not right because it says jpg when there are nef and pgm files % as well a = rd.listArtifacts;
a(1) ans = 

  artifactId: 'RedRose'
 description: ''
   localPath: ''
        name: ''
  remotePath: 'resources/scenes/multiband/scien/2004'
repositoryId: 'isetbio'
        type: 'mat'
         url: [1x100 char]
     version: '1'

%% Fetch the data artifacts s = rd.readArtifact(a(1).artifactId); Error using gradleFetchArtifact (line 100) error status 1 (:fetchIt FAILED

FAILURE: Build failed with an exception.

BUILD FAILED

Total time: 1.094 secs )

benjamin-heasly commented 8 years ago

I think this is a matter of usage / incomplete documentation.

As written above, the read fails because readArtifact() needs to know more than just the artifactId. It also needs to know the remote path where to find an artifact with that id.

Here is a revised version of the first example which should work better:

rdt = RdtClient('isetbio');
rdt.crp('');
artifacts = rdt.searchArtifacts('RedRose', 'type', 'mat');
dataCell = rdt.readArtifacts(artifacts);
data = dataCell{1}

This uses readArtifacts() with an s. This method accepts a struct array of artifact metadata, as returned from searchArtifacts(). It returns a cell array of the same size.

Here is another revised version, which should also read a single artifact. It provides the artifact id and remote path explicitly:

rdt = RdtClient('isetbio');
rdt.crp('');
artifacts = rdt.searchArtifacts('RedRose', 'type', 'mat');
data = rdt.readArtifact(artifacts(1).artifactId, 'remotePath', artifacts(1).remotePath);
benjamin-heasly commented 8 years ago

Here is a revised version of the second example which should work.

rd = RdtClient('isetbio');
rd.crp('/resources/scene');
a = rd.listArtifacts();
s = rd.readArtifact(a(1).artifactId, 'remotePath', a(1).remotePath);

The reason this one failed is that the working remote path /resources/scene is not the full path to the artifact we're trying to read resources/scenes/hyperspectral/harvard_database. So we must supply that path explicitly.

We could equally use readArtifacts() with an s, and pass in a struct array of with one or more search results.

dataCell = rd.readArtifacts(a(1:3));
data = dataCell{3}
benjamin-heasly commented 8 years ago

Another issue was that listArtifacts() was returning only 30 results when the correct number would have been higher.

9b632531eadd537b49088757262f2a2bacd1b9e8 fixes this bug. It raises the default limit from 30 results to 1000 results. It also adds a 'pageSize' parameter for listArtifacts() which allows the caller to specify some other limit.

benjamin-heasly commented 8 years ago

The last issue I see is about artifact types. As above,

% 'type' is not right because it says jpg when there are nef and pgm files as well

This seems related to #41, so I will take it up over there.

wandell commented 8 years ago

I think Ben answered all the questions I raised. Onward.