Closed SPlanzer closed 6 years ago
must handle html responses that are returned in response to the getting of layers
2018-03-16 11:36:00,025 [DEBUG]: connectionpool.py(_make_request:396) >> https://data.linz.govt.nz:443 "GET /services/api/v1/layers/53621/ HTTP/1.1" 504 3541 2018-03-16 11:36:00,026 [INFO]: client.py(_raw_request:168) >> Response: 504 Gateway Time-out in 0:01:30.061388 2018-03-16 11:36:00,026 [DEBUG]: client.py(_raw_request:169) >> Response: headers={'Content-Length': '3541', 'Content-Type': 'text/html', 'Server': 'Koordinates', 'ETag': '"57ce01ec-dd5"', 'Date': 'Thu, 15 Mar 2018 22:35:59 GMT', 'Connection': 'keep-alive'} 2018-03-16 11:36:00,026 [WARNING]: client.py(_raw_request:173) >> Response: 504 Server Error: Gateway Time-out for url: https://data.linz.govt.nz/services/api/v1/layers/53621/: <!DOCTYPE html> <html xmlns:v="urn:schemas-microsoft-com:vml" > <head> <title>Temporarily Unavailable | LINZ Data Service</title> <meta name="description" content="LINZ Data Service - geospatial and related data from Land Information New Zealand. Download GIS data as Shapefile (SHP), FGDB, DWG, MapInfo, CSV, Google Earth (KML)."> <link rel="stylesheet" href="/res/css/release/base.css" type="text/css" media="all"> <style media="all"> .primary_foreground { color: #014B56; } .primary_foreground a { color: #014B56; } .primary_background { background-color: #014B56 !important; } .secondary_foreground { color: #E3E7E8; } .secondary_foreground a { color: #E3E7E8; } .secondary_background { background-color: #E3E7E8; } .midtone_light_foreground { color: #9AB7BC; } .midtone_light_foreground a { color: #9AB7BC; } .midtone_light_background { background-color: #9AB7BC; } .midtone_dark_foreground { color: #669299; } .midtone_dark_foreground a { color: #669299; } .midtone_dark_background { background-color: #669299; } .logo_foreground { color: #014B56; } .logo_background { background-color: #014B56 !important; } .logo_background_border_bottom { border-bottom-color: #014B56 !important; } .highlight_background { background-color: #00778A !important; } .highlight_background:hover { background-color: #008DA3 !important; } /* BUTTONS */ .highlight_background[disabled] { background: #EEE !important; color: #CCC; } .theme_button_inverse { background: #669299; color: #014B56; } .theme_button_inverse a { color: #014B56; } /* Needs to come after theme_button_inverse! */ .theme_button { background: #014B56; color: #9AB7BC; } .theme_button a { color: #9AB7BC; } /* LINKS or CART BUTTON */ .highlight_background.disabled { background-color: inherit !important; } .highlight_background.disabled:hover { background-color: inherit !important; } </style> <style type="text/css" media="all">v\:* {behavior:url(#default#VML);}</style> </head> <body onbeforeunload="" class=" sectionNomap sectionLogin "> <div id="headerLogo" class="headerLogoPosition logo_background"> <a href="https://data.linz.govt.nz" title="LINZ Data Service Home" data-dojo-type="K/widgets/Tooltipify"> <img alt="LINZ Data Service" src="/media/settings/branding/LINZ Data Service_white logo-1-1.png"> </a> </div> <div id="header" class="primary_background secondary_foreground"> <div class="headerInner"></div> </div> <div id="main"> <h1>High load encountered</h1> <p>We're currently experiencing high load. Technical support has been notified and we should be back to normal soon.</p> <p>Please try again shortly, and thank you for your patience.</p> </div> <div id="footer"> <div class="links"> <table> <tr> <td style="text-align:right; padding-right: 10px;"> <strong>LINZ Data Service</strong> </td> <td> <a href='mailto:customersupport@linz.govt.nz'>Contact Us</a> </td> </tr> </table> </div> </div> </body> </html>
I have added a retry in the case of a 504. The the layer still can not be attained this is logger out as an error and raised with the user
https://github.com/linz/lds-metadata-updater/blob/4020bf20e0fdc2b288321bfd265023756370abfb/metadata_updater/metadata_updater.py#L331-L341
must handle html responses that are returned in response to the getting of layers