search

MapStory / mapstory-meta

MapStory
http://mapstory.org
32 stars 16 forks source link

Thumbnail issues #1285

Closed travislbrundage closed 8 years ago

travislbrundage commented 8 years ago

No thumbnail issue causes:

Broken thumbnail issue causes:

garnertb commented 8 years ago

Current broken thumbnail counts:

Layer.objects.filter(thumbnail_url='/static/geonode/img/missing_thumb.png').count()
Map.objects.filter(thumbnail_url='https://mapstory-static.s3.amazonaws.com/geonode/img/missing_thumb.png').count()
garnertb commented 8 years ago

Layers are down to 105 missing thumbnails.

travislbrundage commented 8 years ago

From what I've investigated thus far, it looks like the remaining missing thumbnails have multiple different issues afflicting them. Here's a short list to start.

InvalidOperation: quantize result has too many digits for current context (6 occurrences)

This is a result of the bounding box containing a value which is far too large/too many digits. In all occurrences, one or more elements of the bounding box has a crazy big value. However, this value exists both in GeoServer and in GeoNode for this layer, so it is pulling the data 'correctly' from GeoServer. I have a feeling there is a messed up projection or something bad with this data. I think it might be fine to nuke these since it is just a small handful.

Broken layers and their owners:

Layer oldestUniversities owned by satmaraman
Layer strikes1 owned by hwinscot
Layer Hurricanes by State owned by everett
Layer ArmyInst2 owned by sravaniv
Layer AviationAcc owned by sravaniv
Layer strikes2 owned by hwinscot

FailedRequestError: No store found named: LAYER_NAME (82 occurrences)

This layer does not seem to exist in Geoserver, although it exists in GeoNode. This is the most common error. I believe this probably occurred when the uploader was a bit more finnicky, and it sometimes would "complete" uploading a layer into GeoNode when it hadn't finished loading it into GeoServer, causing a few discrepancies. Perhaps they exist in the database? Need to probe the database still, but if they do not exist there, I don't know how this can be fixed. There may be no choice but to nuke these.

Broken layers and their owners:

Layer Cross_Roads_Replat_A_78_51_Warped_ owned by cadurantobias
Layer Kern_Place_Supplental_Replat_Blk_51_52_23_34_Warped_ owned by cadurantobias
Layer tiftest owned by DanielJDufour
Layer Borderland_Heights_Unit_1_Correction_Plat_70_41_Warped_ owned by cadurantobias
Layer XYZ_2941_ABD owned by hwinscot
Layer _1887 owned by reynalddizon
Layer Crossroads_Subdivision_Unit_2_Replat_A_69_70_Warped_ owned by cadurantobias
Layer mapa_neuquen_georeferenciado owned by erv081
Layer pneumatictransfer owned by nagakuraruan
Layer tonadico7_1856_WGS owned by marco_mastronunzio
Layer napoleonic_1813_WGS owned by marco_mastronunzio
Layer pneumatictransfer0 owned by nagakuraruan
Layer _1_1_2013_DHHP_Homicides_heatmap owned by guybeels
Layer AbrahamBradley_1804 owned by laura.n.eckstein
Layer UVVIS_Clementine_Basemap_v2_ly1 owned by CBiery
Layer BlackDeath_1348 owned by DTauriello
Layer Borderland_Heights_Unit_4_78_71_Warped_ owned by cadurantobias
Layer Boykin_Commercial_District_Unit_4_67_56_Warped_ owned by cadurantobias
Layer Boykin_Commercial_District_Unit_3_65_26_Warped_ owned by cadurantobias
Layer Cross_Roads_Subdivision_8_44_Warped_ owned by cadurantobias
Layer Colonia_Miramonte_Unit_2_55_46_Warped_ owned by cadurantobias
Layer testing owned by guybeels
Layer Frontera_Gardens_1_19_Warped_ owned by cadurantobias
Layer _760 owned by DanielJDufour
Layer Frontera_Gardens_1_19_Warped_0 owned by cadurantobias
Layer Hopkins_DC_1887_plate_26 owned by sshoenfeld
Layer _2014_Bukit_Beras_mag_test_upload owned by Indi
Layer Emory_Park_62_13_Warped_ owned by cadurantobias
Layer _79 owned by DanielJDufour
Layer _1874 owned by BetsyEmmons
Layer Borderland_Heights_Unit_2_29_25_Warped_ owned by cadurantobias
Layer HOLYROMANEMPIRE_1_0 owned by sravaniv
Layer HOLYROMANEMPIRE_1_ owned by sravaniv
Layer IGM_1888_WGS owned by marco_mastronunzio
Layer R_1994_DOQQ owned by ian
Layer Boykin_Commercial_District_Unit_2_52_53A_Warped_ owned by cadurantobias
Layer mideartg20 owned by hwinscot
Layer _76 owned by DanielJDufour
Layer _12FEB27045437_M2AS_R1C1_053282571010_01_P001 owned by WashingtonCollegeGIS
Layer r_1923_2 owned by WashingtonCollegeGIS
Layer Final10 owned by hwinscot
Layer Northwestern Aerial 1975 owned by SeanEmerson
Layer Yosemite Rim Fire - MODIS - 3-day test owned by ryanb
Layer Syria Elevation Model owned by DanielJDufour
Layer Moll_1789_WGS0 owned by marco_mastronunzio
Layer r_1923_3 owned by WashingtonCollegeGIS
Layer Crossroads_Subdivision_Unit_2_62_18_Warped_ owned by cadurantobias
Layer SyriaVegetationZones1 owned by DanielJDufour
Layer SyriaVegetationZones2 owned by DanielJDufour
Layer XYZ_2941_ABC owned by hwinscot
Layer tonadico4_1856_WGS owned by marco_mastronunzio
Layer as_anthrome0 owned by BetsyEmmons
Layer _69 owned by DTauriello
Layer _2012 owned by ryanb
Layer Rim Fire - MODIS - test 2 owned by ryanb
Layer _85 owned by reynalddizon
Layer SyriaVegetationZones owned by DanielJDufour
Layer mideartg2 owned by hwinscot
Layer Borderland_Heights_Unit_1_24_16_Warp3d_ owned by cadurantobias
Layer Frontera_Gardens_No_2_43_9_Warped_ owned by cadurantobias
Layer Crossroads_Commerical_Park_79_83_Warped_ owned by cadurantobias
Layer Colonia_Miramonte_Unit_1_53_30_Warped_ owned by cadurantobias
Layer Dhaka_City_Guide_Map_2003_The_Mappa_Ltd_Sheet_1 owned by SeanEmerson
Layer Frontera_Gardens_Replat_39_15_Warped_ owned by cadurantobias
Layer _10841 owned by sravaniv
Layer Geotiff_LAC_Test_2015 owned by nagakuraruan
Layer wcs owned by tamarasera
Layer _80 owned by reynalddizon
Layer _82 owned by DanielJDufour
Layer Frontera_Sub_77_58_Warped_ owned by cadurantobias
Layer middleearth owned by hwinscot
Layer Colonia_Escondida_41_6_Warped_ owned by cadurantobias
Layer Charl_Ann_Park_55_6_Warped_ owned by cadurantobias
Layer Boykin_Commercial_District_21_6_Warped_ owned by cadurantobias
Layer Andrew Ellicott's plan of Washington D.C. 1791 owned by SeanEmerson
Layer Africa1900 owned by BetsyEmmons
Layer _345 owned by BetsyEmmons
Layer squam_map_4 owned by lycomania
Layer Algonquin Sites on U of R Campus owned by robert.beutner
Layer Colonia_Escondida_Replat_42_15_Warped_ owned by cadurantobias
Layer MOD13A3_2013_12_01.1_km_monthly_NDVI owned by zed.zulkafli.dev
Layer SnowMap owned by BetsyEmmons

Additionally, with the other type of missing thumbnails:

Layer Deal_Island owned by UMD_SAWGAL
Layer Deal_Island0 owned by UMD_SAWGAL
Layer Landfill owned by lmcoll1
Layer Recycled owned by lmcoll1
Layer Bottle owned by lmcoll1
Layer Deal_Island1 owned by UMD_SAWGAL
Layer Book112 owned by MitchGrady

AttributeError: 'NoneType' object has no attribute 'title' (17 occurrences)

In this case, I believe the error occurs due to a mismatch in the stores for the layer in GeoNode vs GeoServer. That is, the layer in GeoNode usually says its store is "geonode"; however, in GeoServer, the layer can be found, but in a different store (not consistently the same one). These may be salvageable by just changing the store.

Something like the following code I believe will fix it:

attribute_layers = []
for layer in Layer.objects.filter(thumbnail_url='/static/geonode/img/missing_thumb.png'):
    try:
        layer.save()
    except AttributeError:
        attribute_layers.append(layer)

for layer in attribute_layers:
    try:
        correct_store = gs_catalog.get_layer(layer.name).resource.store
        gs_catalog.publish_featuretype(layer.name, correct_store, layer.srid)
        layer.store = correct_store
        layer.save()
    # In this scenario, it didn't have a store; known to be 4 cases
    except:
        pass

Of these, 4 of them don't seem to exist in GeoServer at all, so they probably have the same fate as the above layers. They are:

Layer roadl_usa owned by dmfili86
Layer points owned by Kwernsin
Layer points2 owned by mr.daniel.berry
Layer expansion_urbana_2000 owned by Gwendelyn_Monge

Here's all of the errors.

InvalidOperation
Layer: oldestUniversities,
 Layer: strikes1,
 Layer: Hurricanes by State,
 Layer: ArmyInst2,
 Layer: AviationAcc,
 Layer: strikes2

FailedRequestError
Layer: Cross_Roads_Replat_A_78_51_Warped_>,
 Layer: Kern_Place_Supplental_Replat_Blk_51_52_23_34_Warped_>,
 Layer: tiftest>,
 Layer: Borderland_Heights_Unit_1_Correction_Plat_70_41_Warped_>,
 Layer: XYZ_2941_ABD>,
 Layer: _1887>,
 Layer: Crossroads_Subdivision_Unit_2_Replat_A_69_70_Warped_>,
 Layer: mapa_neuquen_georeferenciado>,
 Layer: pneumatictransfer>,
 Layer: tonadico7_1856_WGS>,
 Layer: napoleonic_1813_WGS>,
 Layer: pneumatictransfer0>,
 Layer: _1_1_2013_DHHP_Homicides_heatmap>,
 Layer: AbrahamBradley_1804>,
 Layer: UVVIS_Clementine_Basemap_v2_ly1>,
 Layer: BlackDeath_1348>,
 Layer: Borderland_Heights_Unit_4_78_71_Warped_>,
 Layer: Boykin_Commercial_District_Unit_4_67_56_Warped_>,
 Layer: Boykin_Commercial_District_Unit_3_65_26_Warped_>,
 Layer: Cross_Roads_Subdivision_8_44_Warped_>,
 Layer: Colonia_Miramonte_Unit_2_55_46_Warped_>,
 Layer: testing>,
 Layer: Frontera_Gardens_1_19_Warped_>,
 Layer: _760>,
 Layer: Frontera_Gardens_1_19_Warped_0>,
 Layer: Hopkins_DC_1887_plate_26>,
 Layer: _2014_Bukit_Beras_mag_test_upload>,
 Layer: Emory_Park_62_13_Warped_>,
 Layer: _79>,
 Layer: _1874>,
 Layer: Borderland_Heights_Unit_2_29_25_Warped_>,
 Layer: HOLYROMANEMPIRE_1_0>,
 Layer: HOLYROMANEMPIRE_1_>,
 Layer: IGM_1888_WGS>,
 Layer: R_1994_DOQQ>,
 Layer: Boykin_Commercial_District_Unit_2_52_53A_Warped_>,
 Layer: mideartg20>,
 Layer: _76>,
 Layer: _12FEB27045437_M2AS_R1C1_053282571010_01_P001>,
 Layer: r_1923_2>,
 Layer: Final10>,
 Layer: Northwestern Aerial 1975>,
 Layer: Yosemite Rim Fire - MODIS - 3-day test>,
 Layer: Syria Elevation Model>,
 Layer: Moll_1789_WGS0>,
 Layer: r_1923_3>,
 Layer: Crossroads_Subdivision_Unit_2_62_18_Warped_>,
 Layer: SyriaVegetationZones1>,
 Layer: SyriaVegetationZones2>,
 Layer: XYZ_2941_ABC>,
 Layer: tonadico4_1856_WGS>,
 Layer: as_anthrome0>,
 Layer: _69>,
 Layer: _2012>,
 Layer: Rim Fire - MODIS - test 2>,
 Layer: _85>,
 Layer: SyriaVegetationZones>,
 Layer: mideartg2>,
 Layer: Borderland_Heights_Unit_1_24_16_Warp3d_>,
 Layer: Frontera_Gardens_No_2_43_9_Warped_>,
 Layer: Crossroads_Commerical_Park_79_83_Warped_>,
 Layer: Colonia_Miramonte_Unit_1_53_30_Warped_>,
 Layer: Dhaka_City_Guide_Map_2003_The_Mappa_Ltd_Sheet_1>,
 Layer: Frontera_Gardens_Replat_39_15_Warped_>,
 Layer: _10841>,
 Layer: Geotiff_LAC_Test_2015>,
 Layer: wcs>,
 Layer: _80>,
 Layer: _82>,
 Layer: Frontera_Sub_77_58_Warped_>,
 Layer: middleearth>,
 Layer: Colonia_Escondida_41_6_Warped_>,
 Layer: Charl_Ann_Park_55_6_Warped_>,
 Layer: Boykin_Commercial_District_21_6_Warped_>,
 Layer: Andrew Ellicott's plan of Washington D.C. 1791>,
 Layer: Africa1900>,
 Layer: _345>,
 Layer: squam_map_4>,
 Layer: Algonquin Sites on U of R Campus>,
 Layer: Colonia_Escondida_Replat_42_15_Warped_>,
 Layer: MOD13A3_2013_12_01.1_km_monthly_NDVI>,
 Layer: SnowMap>]

AttributeError
Layer: MLK_speeches0>,
 Layer: Moves>,
 Layer: test_data>,
 Layer: Provinces>,
 Layer: MLK_speeches>,
 Layer: prisonshort>,
 Layer: roadl_usa>,
 Layer: Coph of MLK Speeches>,
 Layer: US_Shootings>,
 Layer: points>,
 Layer: points2>,
 Layer: FANNIE_LOU_HAMER_Sheet2_1_0>,
 Layer: states4>,
 Layer: Birth_Place_Notable_Civil_Rights_Leaders>,
 Layer: OspreyData>,
 Layer: Oil_Deepwater>,
 Layer: expansion_urbana_2000>
travislbrundage commented 8 years ago

Additionally, there are 39 layers with the amazon link to missing thumbnails which save correctly and don't immediately appear to have anything wrong with them, yet have a broken thumbnail. It's possible all that is needed to be done for these is to somehow fix it to regenerate the thumbnail instead of believing it already has one (the amazon static no thumbnail image).

A similar scenario exists for 805 maps with missing thumbnails. 803 of them simply have a bounding box of all 0. To fix them, I believe they must be updated to have a working bounding box, and then it needs to regenerate the thumbnail such that Remote Thumbnail is detected and is the GeoServer WMS request.

It's not a perfect fix since multiple GeoNode layers may exist, changing the bounding box, but I believe this script should be a good enough start for repairing the bounding boxes:

for mapstory in Map.objects.filter(thumbnail_url="/static/geonode/img/missing_thumb.png"):
    if mapstory.bbox_x0 == 0 and mapstory.bbox_x1 == 0 and mapstory.bbox_y0 == 0 and mapstory.bbox_y1 == 0:
        new_bbox = gs_catalog.get_layer(mapstory.local_layers[0].name).resource.native_bbox
        mapstory.bbox_x0 = new_bbox[0]
        mapstory.bbox_y0 = new_bbox[1]
        mapstory.bbox_x1 = new_bbox[2]
        mapstory.bbox_y1 = new_bbox[3]
        mapstory.save()

Of the other 2 maps, one of does not seem to have any layers at all. The map is: Map 2015 Shootings owned by reynalddizon

The other map is unknown, but may fixed. The map is: Map: TestOCT by Efontela2>

Additionally, 59 more maps with the other missing thumbnail image are afflicted with the same issue.

8 of the 10 remaining maps do not seem to have any layers at all. I don't know how that even came to be, but I think these qualify as a "bad data" case and may be best to nuke. They are:

Map Mapping Palestine Stories owned by bettyyu21
Map Testing owned by cd.boundless
Map UE owned by maniek9731
Map test owned by pwitek
Map Sprint Review owned by cd.boundless
Map etb owned by maniek9731
Map Freifunk in Anger-Crottendorf owned by sagie.ohar
Map Standort der Mobilitätsstationen owned by Alexandrafoe

The remaining 2 maps are unknown, but may have been fixed. They are: Map: Larry's Xmas Vacation by ekaba> Map: test3 by Efontela2>

jonpmarino commented 8 years ago

@jj0hns0n fixed most of these. Still some with "No Image". The stock map image should always replace "No Image"