FREVA-CLINT / freva

The Free Evaluation System Framework (FreVa)
Other
10 stars 3 forks source link

add --allow in freva user-data index to allow indexing to continue on error #72

Closed eelucio closed 1 year ago

eelucio commented 1 year ago

added --allow functionality to allow indexing despite folder/file errors

$ freva user-data index --help
usage: freva user-data index [-h] [--data-type {fs}] [--allow] [--debug] [crawl_dir ...]

Index existing user project data to the databrowser

positional arguments:
  crawl_dir             The user directory(s) that needs to be crawled. (default: None)

options:
  -h, --help            show this help message and exit
  --data-type {fs}, --dtype {fs}
                        The data type of the data. (default: fs)
  --allow, -a           Allow indexing to continue on error. (default: False)
  --debug, -v, -d, --verbose
                        Use verbose output. (default: False)

the pytest does not look pretty, probably I can do better...

antarcticrainforest commented 1 year ago

Can you do a git pull main and see if you can resolve the test issues?

eelucio commented 1 year ago

I've got one question over the schema, @antarcticrainforest:

https://github.com/FREVA-CLINT/freva-service-config/compare/6ea79f0b091ba714c51067cf70ee614e77b341a9...5dad019aae32ecba719506ddf3d83764c4f93900

Some the new fields of the schema have multiValued="ture":

  <field name="time" type="rdate" stored="true" indexed="true"/>
  <!-- now lets define extra facets that are not displayd by default. -->
  <field name="mip_era" type="extra_facet" stored="true" indexed="true" multiValued="ture" />
  <field name="activity_id" type="extra_facet" stored="true" indexed="true" multiValued="ture" />
  <field name="institution_id" type="extra_facet" stored="true" indexed="true" multiValued="ture" />
  <field name="source_id" type="extra_facet" stored="true" indexed="true" multiValued="ture" />
  <field name="experiment_id" type="extra_facet" stored="true" indexed="true" multiValued="ture" />
  <field name="member_id" type="extra_facet" stored="true" indexed="true" multiValued="ture" />
  <field name="table_id" type="extra_facet" stored="true" indexed="true" multiValued="ture" />
  <field name="variable_id" type="extra_facet" stored="true" indexed="true" multiValued="ture" />
  <field name="grid_label" type="extra_facet" stored="true" indexed="true" multiValued="ture" />
  <field name="domain" type="extra_facet" stored="true" indexed="true" multiValued="ture" />
  <field name="driving_model" type="extra_facet" stored="true" indexed="true" multiValued="ture" />
  <field name="rcm_name" type="extra_facet" stored="true" indexed="true" multiValued="ture" />
  <field name="rcm_version" type="extra_facet" stored="true" indexed="true" multiValued="ture" />
  <field name="level_type" type="extra_facet" stored="true" indexed="true" multiValued="ture" />
  <field name="time_reduction" type="extra_facet" stored="true" indexed="true" multiValued="ture" />
  <field name="simulation_id" type="extra_facet" stored="true" indexed="true" multiValued="ture" />
  <field name="grid_id" type="extra_facet" stored="true" indexed="true" multiValued="ture" />
  <field name="format" type="extra_facet" stored="true" indexed="true" multiValued="ture" />
</schema>

Is that a typo or is it a way to make them not multiValued?