search

rweigel / datacache

2 stars 1 forks source link

add metadata for cdaweb #7

Closed weiribao closed 11 years ago

weiribao commented 12 years ago

example:

http://aurora.gmu.edu:8000/cache/cdaweb.gsfc.nasa.gov/f80b35c66324764f451e4c81511f9c6b.out

in syncsubmit json result:

metadata: [[EPOCH , <|B|>, ..], [dd-mm-yyyy, ...]] data: [["03-01-2005", "00:00:00.000", ...], [" 03-01-2005"]]

weiribao commented 12 years ago

@rweigel how should the data be stored on file system? Should it be stored as a string like now or in JSON format?

It is weird to store data as unparsed string on file system then parsed every time when requested.

rweigel commented 12 years ago

I think we should store it as md5url.datax.json and md5url.data.json. It should only be recreated if it is not found or if forceUpdate=true.

weiribao commented 12 years ago

OK. That sounds reasonable. Should I create empty json files for other sites besides cdaweb as well?

On Sun, Nov 11, 2012 at 9:21 AM, rweigel notifications@github.com wrote:

I think we should store it as md5url.datax.json and md5url.data.json. It should only be recreated if it is not found or if forceUpdate=true.

— Reply to this email directly or view it on GitHubhttps://github.com/weiribao/tsds_fe/issues/7#issuecomment-10267655.

rweigel commented 12 years ago

I think the logic should be that if includeData is requested, then just insert the raw string of md5url.data unless a data2json method if found for the plugin.

Same logic for includeMetadata.

On Sun, Nov 11, 2012 at 9:26 AM, Bruce Wei notifications@github.com wrote:

OK. That sounds reasonable. Should I create empty json files for other sites besides cdaweb as well?

On Sun, Nov 11, 2012 at 9:21 AM, rweigel notifications@github.com wrote:

I think we should store it as md5url.datax.json and md5url.data.json. It should only be recreated if it is not found or if forceUpdate=true.

— Reply to this email directly or view it on GitHub< https://github.com/weiribao/tsds_fe/issues/7#issuecomment-10267655>.

— Reply to this email directly or view it on GitHubhttps://github.com/weiribao/tsds_fe/issues/7#issuecomment-10267715.

rweigel commented 12 years ago

Seems like the md5url.datax.json and md5url.data.json parts are not being saved - when I do a second request for cached data, I don't get the json string back.

The logic should be

includeData -> includes .data as string unless method exists to parse data. In that case it only returns data as JSON string. includeMetadata -> same as above except for .datax.