Filter settings of "POI" view are not included in the csv export

[Svenyo]

@1000miles @holgerd I have discovered the following bug when trying to do an export of data into a csv file myself.

I go to https://wheelmap.org/admin/pois and set the filters to: category = food node type = any region = Vienna

and click on "Unknown accessibility" in the horizontal menu of the POI list. Now the list only shows the POIS as filtered (btw: looking at the entries here makes me assume this list is not a complete list of POIs existing).

Now, I click on csv to export the filtered data. I expect the list to contain exactly those POIs I see in the list.

What happens instead: I get a csv file of 14,3 MB containing 136,013 lines of (randomized) categories, regions, accessibility stati, ... (I cannot attach it here since csv is not supported).

[holgerd]

I can reproduce this bug:

https://wheelmap.org/admin/pois?utf8=%E2%9C%93&q%5Bnode_type_category_id_eq%5D=2&q%5Bregion_id_eq%5D=68&commit=Filtern&order=osm_id_desc&scope=unknown_accessibility

This is the result when I download the CSV file at the bottom of that page: food-vienna-pois-2016-10-13 (1).csv.zip (143.109 rows, 4MB compressed)

I see POIs from Bengaluru in there and other things which are clearly not in Vienna. This are the first 10 lines:

Id Name Lat Lon Street Housenumber Postcode City Wheelchair Typ Category 4445394521 BigBazzar 129.980.162 77.696.043 Mahadevpura Whitefield Main Road 560048 Bengaluru unknown supermarket shopping 4445392580 VFS Global - Italy Visa Application Centre 5.360.416 -39.927.599 Rue J79 Cocody unknown company misc 4445392579 CIMOD 53.603.079 -39.929.597 unknown company misc 4445392578 All Nations International School 53.604.748 -39.925.775 Rue J79 Cocody unknown school education 4445392577 NEW PUNJABI FOODCORNER 128.475.949 77.677.787 Opposite Wipro Gate 12, Electronic City 560100 Bangalore unknown restaurant food 4445391644 Thoyyab Noorani Chicken And Mutton Shop 129.219.607 776.627.414 Agra, Service Rd, Iblur Village, Bellandur 560103 Bengaluru unknown butcher shopping 4445391106 Columbia Asia Referral Hospital 130.141.709 775.560.212 Brigade Gateway, Beside Metro Cash and Carry West, Malleshwaram 26. Apr 560020 Bangalore unknown hospital health 4445391057 planet Sports 12.991.776 7.757.021 Sampige Road LG-34 560003 bangalore unknown sports shopping 4445390800 École de musique municipale 456.347.716 51.467.379 unknown arts_centre tourism

(encoding errors may come from my crappy Excel import)

@Svenyo Is this blocking #433 ?

[1000miles]

Thanks for reporting, @Svenyo @ @holgerd

Looking at the code we can confirm that currently the csv file does not output the data according to the filter options.

filter options:

• category
• node_type
• region
• version
• tags
• created_at
• updated_at

corresponding filter code snippet:

  filter :category, :as => :select, :collection => proc { Category.all.inject([]){|memo,r| memo << [r.name, r.id]; memo}.sort }
  filter :node_type, :as => :select, :collection => proc { NodeType.all.inject([]){|memo,r| memo << [r.name, r.id]; memo}.sort }
  filter :region, :as => :select, :collection => proc { Region.order('lft ASC').inject([]){|memo,r| memo << ["#{'-' * r.level} #{r.name}", r.id]; memo} }
  filter :version
  filter :tags
  filter :created_at
  filter :updated_at

Source: https://github.com/sozialhelden/wheelmap/blob/master/app/admin/poi.rb#L15-L21

---

csv output:

• id
• name
• lat
• lon
• street
• housenumber
• postcode
• city
• wheelchair
• type
• category

corresponding csv code snippet:

  csv :force_quotes => true do
    column :id
    column :name
    column :lat
    column :lon
    column :street
    column :housenumber
    column :postcode
    column :city
    column :wheelchair
    column :type do |poi|
      poi.node_type.try(:identifier)
    end
    column :category do |poi|
      poi.category.try(:identifier)
    end
  end

Source: https://github.com/sozialhelden/wheelmap/blob/master/app/admin/poi.rb#L45-L61

---

URL: https://wheelmap.org/admin/pois?utf8=✓&q[node_type_category_id_eq]=2&q[region_id_eq]=68&commit=Filtern&scope=unknown_accessibility&order=osm_id_desc&

index columns:

• osm_id
• wheelchair
• version
• name
• node_type
• address
• lat
• lon
• region
• photos

  index do
    column :osm_id do |poi|
      link_to poi.osm_id, node_path(poi)
    end
    column :wheelchair do |poi|
      status_tag(poi.wheelchair, :class => poi.wheelchair)
    end
    column :version
    column :name
    column :node_type
    column :address do |p|
      span p.address
    end
    column :lat
    column :lon
    column :region
    column :photos, :sortable => true do |poi|
      link_to "Photos", admin_poi_photos_path(poi) if poi.photos.size > 0
    end
    actions
  end

Source: https://github.com/sozialhelden/wheelmap/blob/master/app/admin/poi.rb#L63-L83

We will try to optimize this ticket before #433, this way we might be able to provide exporting csv-files via the active admin backend without cumbersome sql database queries. :)

[1000miles]

When trying to download the csv after using the filter options, we get this URL: https://wheelmap.org/admin/pois.csv?order=osm_id_desc&q%5Bnode_type_category_id_eq%5D=2&q%5Bregion_id_eq%5D=68&scope=unknown_accessibility&utf8=%E2%9C%93

[1000miles]

We testet the csv download function on staging.wheelmap.org and wheelmap.org today:

Production:

Kategorie: alle Node Type: Anwalt Region: Berlin

• index page: 64 entries

• Download CSV => It downloads a csv file (9.5 MB) with 4416 lines (with 4415 data entries)

  => not ok!

• CSV queries: "Id","Name","Lat","Lon","Street","Housenumber","Postcode","City","Wheelchair","Typ","Category"

---

Staging:

Kategorie: alle Node Type: Anwalt Region: Berlin

• index page: 40 entries

• Download CSV => It downloads a csv file (5.2kb) with 41 lines (with 40 data entries )

  => works as expected!

• CSV queries: "Id","Name","Lat","Lon","Street","Housenumber","Postcode","City","Wheelchair","Typ","Category"

[1000miles]

Update, 14 oct 2016:

• we checked the gem activeadmin on staging and production, both use the same version
• the Download csv bug in active_admin on production applies to all pages, not to poi pages only
• the last time we deployed to production was 29 days ago, so we will deploy the current state to production on monday and see if the download csv bug has been fixed meanwhile

[1000miles]

Update, 17 oct 2016:

Maybe our csv bug in production is related to this issue and possible solution. We will investigate further if it has to do with ActiveAdmin::CSVBuilder or ActiveRecord::QueryCache

• https://github.com/activeadmin/activeadmin/issues/4118#issuecomment-146245008
• https://github.com/activeadmin/activeadmin/issues/4118#issuecomment-146248353

Investigation steps:

• [x] check if staging server and production use the same active_admin version => both env have same version
• [x] deploy current staging release to production server => did not help, filter for csv export does not work in production
• [x] stop and restart production server => did not help, filter for csv export does not work in production
• [x] research if big data for csv export could be a known issue for active_admin
• [x] load the last recent staging dump to development_env to test it locally (37.500 records, 229 MB)
• [ ] load the last recent production dump to development_env to test it locally

[holgerd]

Maybe related from a year ago: https://github.com/sozialhelden/wheelmap/issues/32

[holgerd]

I suggest we timebox this (eg 4h) and if then re-evaluate if it is worth putting more time into this.

We need to confirm first that CSV exports have business value beyond streetspotr exports in the future. Maybe manual exports are better use of time.

[holgerd]

we discussed we will clarify use cases for the backend system before putting any more work into /admin

So I put this on hold.