limit the number of columns viewed at once in datasource inspector

[springmeyer]

With an osm2pgsql import you can end up with hundreds of columns. It appears that even if I modify tilemill's Datasource.server.bones to only show 100 features at a time, given the massive number of columns this hammers Chroms ability to render the grid. Basically what I'm seeing is that "ChromeRenderer" shows up as pinned at 100% and the grid is un-movable/hung. Chrome in general becomes ultra sluggish and requires a restart to perform well again. Perhaps there is a chrome bug or memory leak, but it seems important at least on the tilemill side to try to limit the # of columns show to something like 50/100.

Also, we currently limit the rows to 1000 - shouldn't this be something more like 200 ?

[springmeyer]

Here is the datasource that leads to this behavior (it is also the one that leads to the crash in #1119):

> ds.describe()
{ type: 'vector',
  extent: 
   [ -8310838.85,
     1977410.68,
     -7592073.62,
     2298890.2 ],
  encoding: 'UTF8',
  fields: 
   { osm_id: 'Number',
     'IOM:project': 'String',
     'IOM_project_DRR:form_number': 'String',
     NOTE: 'String',
     access: 'String',
     'addr:city': 'String',
     'addr:flats': 'String',
     'addr:full': 'String',
     'addr:housenumber': 'String',
     'addr:interpolation': 'String',
     'addr:street': 'String',
     'addr:ward': 'String',
     admin_level: 'String',
     aerialway: 'String',
     aeroway: 'String',
     amenity: 'String',
     'amenity:police': 'String',
     area: 'String',
     attribute_source_date: 'String',
     attribute_source_name: 'String',
     attribute_source_type: 'String',
     barrier: 'String',
     barrier_bypass: 'String',
     'barriers:obstacle_description': 'String',
     bicycle: 'String',
     bladder: 'String',
     borehole: 'String',
     boundary: 'String',
     bridge: 'String',
     'bridge_base_material:concrete': 'String',
     'bridge_base_material:metal': 'String',
     'bridge_base_material:reinforced_concrete': 'String',
     'bridge_base_material:stone': 'String',
     'bridge_base_material:unspecified': 'String',
     'bridge_base_material:wood': 'String',
     bridge_type: 'String',
     building: 'String',
     'building_base_material:canvas': 'String',
     'building_base_material:concrete': 'String',
     'building_base_material:other': 'String',
     'building_base_material:plastic_sheeting': 'String',
     'building_base_material:pvc': 'String',
     'building_base_material:reinforced_concrete': 'String',
     'building_base_material:sheet_metal': 'String',
     'building_base_material:tent': 'String',
     'building_base_material:unspecified': 'String',
     'building_base_material:wood': 'String',
     'building_roof_material:canvas': 'String',
     'building_roof_material:concrete': 'String',
     'building_roof_material:other': 'String',
     'building_roof_material:plastic_sheeting': 'String',
     'building_roof_material:pvc': 'String',
     'building_roof_material:sheet_metal': 'String',
     'building_roof_material:stone': 'String',
     'building_roof_material:tent': 'String',
     'building_roof_material:tiles': 'String',
     'building_roof_material:unspecified': 'String',
     'building_roof_material:wood': 'String',
     building_status: 'String',
     cafeteria: 'String',
     'canal:obstacle_type': 'String',
     canal_banks_barriers_desc_left: 'String',
     canal_banks_barriers_desc_right: 'String',
     canal_banks_barriers_left: 'String',
     canal_banks_barriers_right: 'String',
     canal_banks_desc_left: 'String',
     canal_banks_desc_right: 'String',
     canal_banks_left: 'String',
     canal_banks_right: 'String',
     canal_base_material: 'String',
     canal_description: 'String',
     canal_roof_desc: 'String',
     canal_roof_extent: 'String',
     canal_type: 'String',
     capacity: 'String',
     'capacity:beds': 'String',
     'capacity:seats': 'String',
     'capacity:standing': 'String',
     cholera_affected_people: 'String',
     'classrooms:afternoon': 'String',
     'classrooms:evening': 'String',
     'classrooms:morning': 'String',
     'classrooms:night': 'String',
     clinic: 'String',
     construction: 'String',
     cutting: 'String',
     depth: 'String',
     disability_access: 'String',
     disused: 'String',
     doctors: 'String',
     drinking_water: 'String',
     embankment: 'String',
     'employees:afternoon': 'String',
     'employees:evening': 'String',
     'employees:morning': 'String',
     'employees:night': 'String',
     employees_non_medical: 'String',
     first_aid: 'String',
     flood_prone: 'String',
     'flood_prone:first_year': 'String',
     'flood_prone:frequency': 'String',
     'flood_prone:height': 'String',
     'flood_prone:impact': 'String',
     'flood_prone:informant': 'String',
     'flood_prone:rise': 'String',
     'flood_prone:water_clear': 'String',
     'flood_prone:water_detritus': 'String',
     'flood_prone:water_mud': 'String',
     foot: 'String',
     gender_access: 'String',
     'generator:source': 'String',
     'generator:source_others': 'String',
     geometry_source_date: 'String',
     geometry_source_name: 'String',
     geometry_source_type: 'String',
     health_facility_type: 'String',
     'healthcare:cholera': 'String',
     'healthcare_departments:ambulances': 'String',
     'healthcare_departments:general_medicine': 'String',
     'healthcare_departments:medical_laboratory': 'String',
     'healthcare_departments:operating_room': 'String',
     'healthcare_departments:others': 'String',
     highway: 'String',
     historic: 'String',
     horse: 'String',
     'id:db_ctc': 'String',
     'id:uid': 'String',
     'id:uuid': 'String',
     infirmary: 'String',
     junction: 'String',
     kitchen: 'String',
     landfill_authority: 'String',
     landfill_authority_other: 'String',
     landfill_height: 'String',
     landfill_length: 'String',
     'landfill_type:dumpster': 'String',
     'landfill_type:ground': 'String',
     'landfill_type:other': 'String',
     'landfill_type:pit': 'String',
     landfill_width: 'String',
     landuse: 'String',
     layer: 'String',
     learning: 'String',
     leisure: 'String',
     length: 'String',
     lock: 'String',
     man_made: 'String',
     'marketplace:alimentary': 'String',
     'marketplace:authority': 'String',
     'marketplace:covered': 'String',
     'marketplace:note': 'String',
     'marketplace:open_air': 'String',
     'marketplace:specialized': 'String',
     'marketplace:stands': 'String',
     marketplace_status: 'String',
     maxweight: 'String',
     military: 'String',
     motorcar: 'String',
     name: 'String',
     'name:en': 'String',
     'name:fr': 'String',
     'name:kr': 'String',
     natural: 'String',
     oneway: 'String',
     'opening_hours:afternoon': 'String',
     'opening_hours:evening': 'String',
     'opening_hours:friday': 'String',
     'opening_hours:monday': 'String',
     'opening_hours:morning': 'String',
     'opening_hours:night': 'String',
     'opening_hours:saturday': 'String',
     'opening_hours:sunday': 'String',
     'opening_hours:thursday': 'String',
     'opening_hours:tuesday': 'String',
     'opening_hours:unspecified': 'String',
     'opening_hours:wednesday': 'String',
     operational_status: 'String',
     operational_status_quality: 'String',
     operator: 'String',
     operator_id: 'String',
     operator_type: 'String',
     operator_type_other: 'String',
     organization_location_type: 'String',
     osm_timestamp: 'String',
     osm_uid: 'String',
     osm_user: 'String',
     osm_version: 'String',
     photo_name: 'String',
     photo_num: 'String',
     photo_url: 'String',
     pitlatrine: 'String',
     place: 'String',
     'place:area': 'String',
     'place:area_alternate': 'String',
     'place:description': 'String',
     'place:nearestlandmark': 'String',
     power: 'String',
     power_source: 'String',
     practicability: 'String',
     'public_adults:afternoon': 'String',
     'public_adults:evening': 'String',
     'public_adults:morning': 'String',
     'public_adults:night': 'String',
     'public_under18:afternoon': 'String',
     'public_under18:evening': 'String',
     'public_under18:morning': 'String',
     'public_under18:night': 'String',
     'pupils:afternoon': 'String',
     'pupils:evening': 'String',
     'pupils:morning': 'String',
     'pupils:night': 'String',
     railway: 'String',
     ramp: 'String',
     ref: 'String',
     registry_name: 'String',
     religion: 'String',
     'religion:denomination': 'String',
     residence: 'String',
     roof_status: 'String',
     route: 'String',
     'school:first_cycle': 'String',
     'school:preschool_cycle': 'String',
     'school:second_cycle': 'String',
     'school:third_cycle': 'String',
     service: 'String',
     shop: 'String',
     sport: 'String',
     status: 'String',
     store: 'String',
     surface: 'String',
     surface_condition: 'String',
     survey_information_access: 'String',
     'surveyor:GPS': 'String',
     'surveyor:email': 'String',
     'surveyor:name': 'String',
     'surveyor:phone': 'String',
     tank: 'String',
     tap: 'String',
     'telecom:internet': 'String',
     'telecom:land_line': 'String',
     'telecom:radio': 'String',
     'toilets:access': 'String',
     'toilets:basement': 'String',
     'toilets:cleaning': 'String',
     'toilets:durability': 'String',
     'toilets:emptying': 'String',
     'toilets:tank': 'String',
     'toilets:technology': 'String',
     tourism: 'String',
     tracktype: 'String',
     transportation_use_category: 'String',
     tunnel: 'String',
     'waste_type:building_waste': 'String',
     'waste_type:common_urban_waste': 'String',
     'waste_type:industrial_polluting': 'String',
     'waste_type:other': 'String',
     'water_place:access': 'String',
     'water_place:cleaning': 'String',
     'water_place:durability': 'String',
     water_purification: 'String',
     water_supply: 'String',
     waterway: 'String',
     way_area: 'String',
     width: 'String',
     wood: 'String',
     z_order: 'Number',
     gid: 'Number' },
  geometry_type: 'linestring',
  has_features: true }

[yhahn]

Ha, yeah. We can hard clip this with the expectation that the user should limit the number of columns they query for?

Would 20 be reasonable? /cc @dhcole @ajashton

[dhcole]

yes, I think 20 is good.

[dhcole]

@smit1678 what do you think?

[springmeyer]

I was hoping for a "show more" button, but ya, if that seems insane hard clipping would work, though I worry that would make it hard to know what was being hidden for osm users.

[springmeyer]

Another approach could be to count the # of columns, if if there are 100, only show 50 rows at first, or if there are 200 only show 30 rows at first, etc.

[yhahn]

I went with 50 columns with a message about omitted fields as some of our default mapbox geodata shapefiles have just over 30 columns. Let me know if this fixes the UI slowdown for you.

[willwhite]

This used to limited by number of cells, so the more columns there were, the fewer rows would be displayed.

[springmeyer]

On further thought, I'm feeling quite uncomfortable with a hard limit on 50 cols. This is going to confuse OSM users trying to learn about their data. @willwhite - are you indicating this used to be more performant (less rows were shown previously) but something regressed?

[yhahn]

I believe a cell-based limit was quite confusing as well.

Currently any time the columns are truncated the following message is displayed:

ISO_A2, ISO_A3, NAME fields omitted. Use fewer fields for better performance.

Still too confusing?

[yhahn]

I'm going to close this for now... i think what we have should give users enough information to drop/rewrite their query in a way to explore the omitted fields and exclude ones that they aren't interested in.