Feature Request - remove uncached attributes from search results

[dustymc]

Is your feature request related to a problem? Please describe.

Attributes are expensive, people like clicking all the buttons, that plugs up the toobs.

Describe what you're trying to accomplish

Less-frustrating user experience

Describe the solution you'd like

Remove non-cached attributes from search result options. They're still available as Attribute Detail and via the tools menu.

Describe alternatives you've considered

Maybe we could somehow throw an error when more than some number of uncached attributes are selected or something of the sort, seems like lots of confusion and complexity for something that's probably not actually wanted and is readily available in other forms.

Additional context

See also https://github.com/ArctosDB/arctos/discussions/6179, this isn't unexpected.

From the logs - a limit of 3 records is still timing out.

create table temp_cache.api_pubusrallall_20231103111103800_771 AS SELECT distinct filtered_flat.collection_object_id,filtered_flat.guid as guid,filtered_flat.scientific_name as scientific_name,filtered_flat.country as country,filtered_flat.state_prov as state_prov,filtered_flat.spec_locality as spec_locality,filtered_flat.verbatim_date as verbatim_date,filtered_flat.dec_lat as dec_lat,filtered_flat.dec_long as dec_long,filtered_flat.coordinateuncertaintyinmeters as coordinateuncertaintyinmeters,filtered_flat.species as species,filtered_flat.island as island,filtered_flat.min_elev_in_m as min_elev_in_m,filtered_flat.max_elev_in_m as max_elev_in_m,filtered_flat.minimum_elevation as minimum_elevation,filtered_flat.maximum_elevation as maximum_elevation,filtered_flat.orig_elev_units as orig_elev_units,case when filtered_flat.MIN_ELEV_IN_M is null then null else filtered_flat.MIN_ELEV_IN_M||'-'|| filtered_flat.MAX_ELEV_IN_M end as elev_in_m,filtered_flat.habitat as habitat,concatAttributeValue(filtered_flat.collection_object_id,'air temperature') as air_temperature,concatAttributeValue(filtered_flat.collection_object_id,'fat deposition') as fat_deposition,filtered_flat.sex as sex,filtered_flat.weight as weight,filtered_flat.catalognumberint as catalognumberint,filtered_flat.identifiers::varchar as identifiers,filtered_flat.othercatalognumbers as othercatalognumbers,filtered_flat.ACCESSION as accn_number,filtered_flat.media::varchar as media,filtered_flat.identifiedby as identified_by,filtered_flat.id_sensu as id_sensu,filtered_flat.made_date as id_date,filtered_flat.formatted_scientific_name as sci_name_with_auth,filtered_flat.previousIdentifications::varchar as id_history,filtered_flat.identification_remarks as identification_remarks,filtered_flat.made_date as made_date,filtered_flat.collection_object_id as remove_row,filtered_flat.kingdom as kingdom,filtered_flat.phylum as phylum,filtered_flat.phylclass as phylclass,filtered_flat.superorder as superorder,filtered_flat.phylorder as phylorder,filtered_flat.suborder as suborder,filtered_flat.superfamily as superfamily,filtered_flat.family as family,filtered_flat.subfamily as subfamily,filtered_flat.tribe as tribe,filtered_flat.subtribe as subtribe,filtered_flat.genus as genus,filtered_flat.subspecies as subspecies,filtered_flat.typestatus as citations,filtered_flat.relatedcatalogeditems as relatedcatalogeditems,filtered_flat.collectors as collectors,filtered_flat.preparators as preparators,filtered_flat.creator as creator,filtered_flat.subject as subject,filtered_flat.copyright_holder as copyright_holder,filtered_flat.SPECIMEN_EVENT_TYPE as specimen_event_type,filtered_flat.EVENT_ASSIGNED_BY_AGENT as event_assigned_by_agent,to_char(filtered_flat.EVENT_ASSIGNED_DATE,'yyyy-mm-dd') as event_assigned_date,filtered_flat.collecting_source as collecting_source,filtered_flat.SPECIMEN_EVENT_REMARK as specimen_event_remark,filtered_flat.continent_ocean as continent_ocean,filtered_flat.sea as sea,filtered_flat.quad as quad,filtered_flat.feature as feature,filtered_flat.county as county,filtered_flat.island_group as island_group,case when filtered_flat.MIN_DEPTH_IN_M is null then null else filtered_flat.MIN_DEPTH_IN_M||'-'|| filtered_flat.MAX_DEPTH_IN_M end as depth_in_m,filtered_flat.locality_remarks as locality_remarks,filtered_flat.verbatim_locality as verbatim_locality,filtered_flat.LOCALITY_NAME as locality_name,filtered_flat.collecting_method as collecting_method,filtered_flat.began_date as began_date,filtered_flat.ended_date as ended_date,getDayCollected(filtered_flat.began_date,filtered_flat.ended_date) as day_collected,getMonthCollected(filtered_flat.began_date,filtered_flat.ended_date) as month_collected,filtered_flat.year as year_collected,filtered_flat.parts as parts,filtered_flat.partdetail::varchar as partdetail,case strpos(filtered_flat.parts,'skull') when 0 then 'false' else 'true' end as skull_yn,filtered_flat.remarks as remark,filtered_flat.collecting_event_id as collecting_event_id,filtered_flat.VERBATIM_COORDINATES as verbatim_coordinates,filtered_flat.orig_lat_long_units as orig_lat_long_units,filtered_flat.verificationstatus as verificationstatus,filtered_flat.datum as datum,filtered_flat.GEOREFERENCE_PROTOCOL as georeference_protocol,filtered_flat.GEOREFERENCE_SOURCE as georeference_source,filtered_flat.individualcount as individualcount,filtered_flat.formatted_scientific_name as formatted_scientific_name,filtered_flat.COLL_EVENT_REMARKS as coll_event_remarks,filtered_flat.associated_species as associated_species,getSpecimenEventDeterminer(filtered_flat.collection_object_id) as specimeneventdeterminer,CONCATTYPESTATUS_FULLPUB(filtered_flat.collection_object_id) as long_citation,filtered_flat.json_locality::varchar as json_locality,filtered_flat.formation as formation,filtered_flat.member as member,filtered_flat.event_count as event_count,filtered_flat.locality_count as locality_count,filtered_flat.full_taxon_name as full_taxon_name,filtered_flat.enteredby as entered_by,filtered_flat.related_record_cache::varchar as related_record_summary,filtered_flat.attributedetail::varchar as attributedetail,filtered_flat.LastPartLocation as lastpartlocation,concatAttributeValue(filtered_flat.collection_object_id,'abundance') as abundance,filtered_flat.age as age,filtered_flat.age_class as age_class,concatAttributeValue(filtered_flat.collection_object_id,'archaeological feature') as archaeological_feature,concatAttributeValue(filtered_flat.collection_object_id,'axillary girth') as axillary_girth,concatAttributeValue(filtered_flat.collection_object_id,'behavior') as behavior,concatAttributeValue(filtered_flat.collection_object_id,'bill depth') as bill_depth,concatAttributeValue(filtered_flat.collection_object_id,'bill length') as bill_length,concatAttributeValue(filtered_flat.collection_object_id,'bill width') as bill_width,concatAttributeValue(filtered_flat.collection_object_id,'body condition') as body_condition,concatAttributeValue(filtered_flat.collection_object_id,'body temperature') as body_temperature,concatAttributeValue(filtered_flat.collection_object_id,'body width') as body_width,concatAttributeValue(filtered_flat.collection_object_id,'breadth') as breadth,concatAttributeValue(filtered_flat.collection_object_id,'brood patch') as brood_patch,concatAttributeValue(filtered_flat.collection_object_id,'bursa') as bursa,concatAttributeValue(filtered_flat.collection_object_id,'bursa length') as bursa_length,concatAttributeValue(filtered_flat.collection_object_id,'bursa width') as bursa_width,concatAttributeValue(filtered_flat.collection_object_id,'calcar length') as calcar_length,concatAttributeValue(filtered_flat.collection_object_id,'carapace length') as carapace_length,concatAttributeValue(filtered_flat.collection_object_id,'caste') as caste,concatAttributeValue(filtered_flat.collection_object_id,'chromatography compound') as chromatography_compound,concatAttributeValue(filtered_flat.collection_object_id,'clutch size') as clutch_size,concatAttributeValue(filtered_flat.collection_object_id,'clutch size of nest parasite') as clutch_size_of_nest_parasite,concatAttributeValue(filtered_flat.collection_object_id,'colors') as colors,concatAttributeValue(filtered_flat.collection_object_id,'copyright status') as copyright_status,concatAttributeValue(filtered_flat.collection_object_id,'credit line') as credit_line,concatAttributeValue(filtered_flat.collection_object_id,'crop contents') as crop_contents,concatAttributeValue(filtered_flat.collection_object_id,'crown-rump length') as crown_rump_length,concatAttributeValue(filtered_flat.collection_object_id,'crystal habit') as crystal_habit,concatAttributeValue(filtered_flat.collection_object_id,'culmen length') as culmen_length,filtered_flat.culture_of_origin as culture_of_origin,filtered_flat.culture_of_use as culture_of_use,concatAttributeValue(filtered_flat.collection_object_id,'curved carapace length') as curved_carapace_length,concatAttributeValue(filtered_flat.collection_object_id,'curvilinear length') as curvilinear_length,concatAttributeValue(filtered_flat.collection_object_id,'description') as description,concatAttributeValue(filtered_flat.collection_object_id,'dimensions') as dimensions,concatAttributeValue(filtered_flat.collection_object_id,'diploid number') as diploid_number,concatAttributeValue(filtered_flat.collection_object_id,'ear from crown') as ear_from_crown,filtered_flat.ear_from_notch as ear_from_notch,concatAttributeValue(filtered_flat.collection_object_id,'egg content weight') as egg_content_weight,concatAttributeValue(filtered_flat.collection_object_id,'eggshell thickness') as eggshell_thickness,concatAttributeValue(filtered_flat.collection_object_id,'electrical conductivity') as electrical_conductivity,concatAttributeValue(filtered_flat.collection_object_id,'exhibit caption') as exhibit_caption,concatAttributeValue(filtered_flat.collection_object_id,'experimental') as experimental,concatAttributeValue(filtered_flat.collection_object_id,'extension') as extension,concatAttributeValue(filtered_flat.collection_object_id,'forearm length') as forearm_length,concatAttributeValue(filtered_flat.collection_object_id,'fork length') as fork_length,concatAttributeValue(filtered_flat.collection_object_id,'framed') as framed,concatAttributeValue(filtered_flat.collection_object_id,'gape width') as gape_width,concatAttributeValue(filtered_flat.collection_object_id,'gonad') as gonad,concatAttributeValue(filtered_flat.collection_object_id,'gonad weight') as gonad_weight,concatAttributeValue(filtered_flat.collection_object_id,'hallux length') as hallux_length,concatAttributeValue(filtered_flat.collection_object_id,'head length') as head_length,concatAttributeValue(filtered_flat.collection_object_id,'head width') as head_width,concatAttributeValue(filtered_flat.collection_object_id,'height') as height,filtered_flat.hind_foot_with_claw as hind_foot_with_claw,concatAttributeValue(filtered_flat.collection_object_id,'hind foot without claw') as hind_foot_without_claw,concatAttributeValue(filtered_flat.collection_object_id,'hind limb length') as hind_limb_length,concatAttributeValue(filtered_flat.collection_object_id,'historical') as historical,concatAttributeValue(filtered_flat.collection_object_id,'image confirmed') as image_confirmed,concatAttributeValue(filtered_flat.collection_object_id,'incubation stage') as incubation_stage,concatAttributeValue(filtered_flat.collection_object_id,'individual count') as individual_count,concatAttributeValue(filtered_flat.collection_object_id,'inscriptions and marks') as inscriptions_and_marks,concatAttributeValue(filtered_flat.collection_object_id,'isotope value') as isotope_value,concatAttributeValue(filtered_flat.collection_object_id,'keywords') as keywords,concatAttributeValue(filtered_flat.collection_object_id,'left gonad length') as left_gonad_length,concatAttributeValue(filtered_flat.collection_object_id,'left gonad width') as left_gonad_width,concatAttributeValue(filtered_flat.collection_object_id,'life stage') as life_stage,concatAttributeValue(filtered_flat.collection_object_id,'location in host') as location_in_host,concatAttributeValue(filtered_flat.collection_object_id,'luster') as luster,filtered_flat.materials as materials,concatAttributeValue(filtered_flat.collection_object_id,'maximum standard length') as maximum_standard_length,concatAttributeValue(filtered_flat.collection_object_id,'maximum total length') as maximum_total_length,concatAttributeValue(filtered_flat.collection_object_id,'middle toe length') as middle_toe_length,concatAttributeValue(filtered_flat.collection_object_id,'minimum standard length') as minimum_standard_length,concatAttributeValue(filtered_flat.collection_object_id,'minimum total length') as minimum_total_length,concatAttributeValue(filtered_flat.collection_object_id,'molt condition') as molt_condition,concatAttributeValue(filtered_flat.collection_object_id,'NAGPRA category') as nagpra_category,concatAttributeValue(filtered_flat.collection_object_id,'neck width') as neck_width,concatAttributeValue(filtered_flat.collection_object_id,'nest description') as nest_description,concatAttributeValue(filtered_flat.collection_object_id,'nest phenology') as nest_phenology,concatAttributeValue(filtered_flat.collection_object_id,'number of labels') as number_of_labels,concatAttributeValue(filtered_flat.collection_object_id,'numeric age') as numeric_age,concatAttributeValue(filtered_flat.collection_object_id,'object title') as object_title,concatAttributeValue(filtered_flat.collection_object_id,'ovum') as ovum,concatAttributeValue(filtered_flat.collection_object_id,'parts examined') as parts_examined,concatAttributeValue(filtered_flat.collection_object_id,'pit depth') as pit_depth,concatAttributeValue(filtered_flat.collection_object_id,'plastron length') as plastron_length,concatAttributeValue(filtered_flat.collection_object_id,'portfolio or series') as portfolio_or_series,concatAttributeValue(filtered_flat.collection_object_id,'processing history') as processing_history,concatAttributeValue(filtered_flat.collection_object_id,'provenience') as provenience,concatAttributeValue(filtered_flat.collection_object_id,'provenience east') as provenience_east,concatAttributeValue(filtered_flat.collection_object_id,'provenience north') as provenience_north,concatAttributeValue(filtered_flat.collection_object_id,'quadrant') as quadrant,concatAttributeValue(filtered_flat.collection_object_id,'radiometric date') as radiometric_date,concatAttributeValue(filtered_flat.collection_object_id,'reproductive data') as reproductive_data,concatAttributeValue(filtered_flat.collection_object_id,'right gonad length') as right_gonad_length,concatAttributeValue(filtered_flat.collection_object_id,'right gonad width') as right_gonad_width,concatAttributeValue(filtered_flat.collection_object_id,'skull ossification') as skull_ossification,concatAttributeValue(filtered_flat.collection_object_id,'snout-vent length') as snout_vent_length,concatAttributeValue(filtered_flat.collection_object_id,'SNV results') as snv_results,concatAttributeValue(filtered_flat.collection_object_id,'soft parts') as soft_parts,concatAttributeValue(filtered_flat.collection_object_id,'spleen length') as spleen_length,concatAttributeValue(filtered_flat.collection_object_id,'square') as square,concatAttributeValue(filtered_flat.collection_object_id,'standard length') as standard_length,concatAttributeValue(filtered_flat.collection_object_id,'stomach contents') as stomach_contents,concatAttributeValue(filtered_flat.collection_object_id,'straight carapace length') as straight_carapace_length,concatAttributeValue(filtered_flat.collection_object_id,'stratigraphic position') as stratigraphic_position,concatAttributeValue(filtered_flat.collection_object_id,'tail base width') as tail_base_width,concatAttributeValue(filtered_flat.collection_object_id,'tail condition') as tail_condition,concatAttributeValue(filtered_flat.collection_object_id,'tail girth') as tail_girth,filtered_flat.tail_length as tail_length,concatAttributeValue(filtered_flat.collection_object_id,'tarsus length') as tarsus_length,concatAttributeValue(filtered_flat.collection_object_id,'tested for presence') as tested_for_presence,concatAttributeValue(filtered_flat.collection_object_id,'thumb length with claw') as thumb_length_with_claw,concatAttributeValue(filtered_flat.collection_object_id,'tooth length') as tooth_length,concatAttributeValue(filtered_flat.collection_object_id,'tooth width') as tooth_width,filtered_flat.total_length as total_length,concatAttributeValue(filtered_flat.collection_object_id,'tragus length') as tragus_length,concatAttributeValue(filtered_flat.collection_object_id,'trap type') as trap_type,concatAttributeValue(filtered_flat.collection_object_id,'unformatted measurements') as unformatted_measurements,concatAttributeValue(filtered_flat.collection_object_id,'value') as value,concatAttributeValue(filtered_flat.collection_object_id,'verbatim agent') as verbatim_agent,concatAttributeValue(filtered_flat.collection_object_id,'verbatim host age') as verbatim_host_age,concatAttributeValue(filtered_flat.collection_object_id,'verbatim host ID') as verbatim_host_id,concatAttributeValue(filtered_flat.collection_object_id,'verbatim host sex') as verbatim_host_sex,concatAttributeValue(filtered_flat.collection_object_id,'verbatim preservation date') as verbatim_preservation_date,concatAttributeValue(filtered_flat.collection_object_id,'water temperature') as water_temperature,concatAttributeValue(filtered_flat.collection_object_id,'width') as width,concatAttributeValue(filtered_flat.collection_object_id,'wing chord') as wing_chord,concatAttributeValue(filtered_flat.collection_object_id,'wing span') as wing_span,concatAttributeValue(filtered_flat.collection_object_id,'year class') as year_class,concatLocalityAttributeValue(filtered_flat.locality_id,'feature') as locality_feature,filtered_flat.af as AF,filtered_flat.collectornumber as collector_number,filtered_flat.genbanknum as GenBank,filtered_flat.nk as NK,filtered_flat.organism_id as Organism_ID,filtered_flat.preparatornumber as preparator_number,filtered_flat.examined_for as examined_for,filtered_flat.not_examined_for as not_examined_for,filtered_flat.detected as detected,filtered_flat.not_detected as not_detected from filtered_flat INNER JOIN cataloged_item ON filtered_flat.collection_object_id = cataloged_item.collection_object_id inner join identification ON filtered_flat.collection_object_id = identification.collection_object_id inner join identification_taxonomy ON identification.identification_id = identification_taxonomy.identification_id INNER JOIN common_name ON (identification_taxonomy.taxon_name_id = common_name.taxon_name_id) WHERE filtered_flat.collection_object_id IS NOT NULL and UPPER(common_name.Common_Name) like '% VOLE%' limit 3

Priority

High, this must be frustrating UX and I believe may be causing some instability.

[campmlc]

Do we have documentation to explain to people how to search on attributes? Will this prevent people from requesting attributes via an Async Request? How to you request values to be downloaded if they don't appear in a visible table to select from? Do we need a basic search template for attributes or "traits"?

Currently it is almost impossible to find and download the rich amount of attribute data that we have in Arctos. I'm not saying to not remove non-cached values, but we need clear, usable alternate tools, and I don't have any idea what those are or how to direct people to use them. I would rather have those tools in place before we continue to reduce discoverability.

A couple of things that would help would be to have an attribute dropdown part of the default search template. Another would be to add "Attribute summary" to the results in the same way we have a parts summary and parts detail. I believe this was previously requested but never acted upon.

[dustymc]

See error logs btime=2023-11-08%2016:55&etime=2023-11-08%2018:00 someone is having a very bad day because they've turned everything on, @mkoo thoughts on proceeding or discussing or whatever it takes to not get stuck?

[dustymc]

Testing https://github.com/ArctosDB/arctos/issues/6201 is causing timeouts in the logs. I feel like we have to act on this, but I don't want to cut "us" off from important data (which might not be cached). HELP!!!!

[mkoo]

Turn uncached attributes off in the results. this is about expectations-- these data are available through many other tools. thanks!

[dustymc]

Pre-update stash:

cf_cat_rec_rslt_cols.csv.zip