linz / lds-metadata-updater

For downloading, editing and then setting LDS metadata
0 stars 2 forks source link

Handle html response to the getting of layers #15

Closed SPlanzer closed 6 years ago

SPlanzer commented 6 years ago

must handle html responses that are returned in response to the getting of layers

2018-03-16 11:36:00,025 [DEBUG]: connectionpool.py(_make_request:396) >> https://data.linz.govt.nz:443 "GET /services/api/v1/layers/53621/ HTTP/1.1" 504 3541
2018-03-16 11:36:00,026 [INFO]: client.py(_raw_request:168) >> Response: 504 Gateway Time-out in 0:01:30.061388
2018-03-16 11:36:00,026 [DEBUG]: client.py(_raw_request:169) >> Response: headers={'Content-Length': '3541', 'Content-Type': 'text/html', 'Server': 'Koordinates', 'ETag': '"57ce01ec-dd5"', 'Date': 'Thu, 15 Mar 2018 22:35:59 GMT', 'Connection': 'keep-alive'}
2018-03-16 11:36:00,026 [WARNING]: client.py(_raw_request:173) >> Response: 504 Server Error: Gateway Time-out for url: https://data.linz.govt.nz/services/api/v1/layers/53621/: <!DOCTYPE html>
<html xmlns:v="urn:schemas-microsoft-com:vml" >
    <head>

        <title>Temporarily Unavailable | LINZ Data Service</title>
        <meta name="description" content="LINZ Data Service - geospatial and related data from Land Information New Zealand. Download GIS data as Shapefile (SHP), FGDB, DWG, MapInfo, CSV, Google Earth (KML).">
        <link rel="stylesheet" href="/res/css/release/base.css" type="text/css" media="all">
        <style media="all">

.primary_foreground {
    color: #014B56;
}
.primary_foreground a {
    color: #014B56;
}
.primary_background {
    background-color: #014B56 !important;
}
.secondary_foreground {
    color: #E3E7E8;
}
.secondary_foreground a {
    color: #E3E7E8;   
}
.secondary_background {
    background-color: #E3E7E8;
}
.midtone_light_foreground {
    color: #9AB7BC;
}
.midtone_light_foreground a {
    color: #9AB7BC;
}
.midtone_light_background {
    background-color: #9AB7BC;
}
.midtone_dark_foreground {
    color: #669299;
}
.midtone_dark_foreground a {
    color: #669299;
}
.midtone_dark_background {
    background-color: #669299;
}
.logo_foreground {
    color: #014B56;
}
.logo_background {
    background-color: #014B56 !important;
}
.logo_background_border_bottom {
    border-bottom-color: #014B56 !important;
}
.highlight_background {
    background-color: #00778A !important;
}
.highlight_background:hover {
    background-color: #008DA3 !important;
}
/* BUTTONS */
.highlight_background[disabled] {
    background: #EEE !important;
    color: #CCC;
}
.theme_button_inverse {
    background: #669299;
    color: #014B56;
}
.theme_button_inverse a {
    color: #014B56;
}
/* Needs to come after theme_button_inverse! */
.theme_button {
    background: #014B56;
    color: #9AB7BC;
}
.theme_button a {
    color: #9AB7BC;
}

/* LINKS or CART BUTTON */
.highlight_background.disabled {
    background-color: inherit !important;
}
.highlight_background.disabled:hover {
    background-color: inherit !important;
}

        </style>
        <style type="text/css" media="all">v\:* {behavior:url(#default#VML);}</style>

    </head>
    <body onbeforeunload="" class=" sectionNomap sectionLogin ">

<div id="headerLogo" class="headerLogoPosition logo_background">
    <a href="https://data.linz.govt.nz" title="LINZ Data Service Home" data-dojo-type="K/widgets/Tooltipify">

            <img alt="LINZ Data Service" src="/media/settings/branding/LINZ Data Service_white logo-1-1.png">

    </a>
</div>

        <div id="header" class="primary_background secondary_foreground">
            <div class="headerInner"></div>
        </div>

        <div id="main">

    <h1>High load encountered</h1>
    <p>We're currently experiencing high load. Technical support has been notified and we should be back to normal soon.</p>
    <p>Please try again shortly, and thank you for your patience.</p>

         </div>

        <div id="footer">
            <div class="links">
                <table>

                    <tr>
                        <td style="text-align:right; padding-right: 10px;">
                            <strong>LINZ Data Service</strong>
                        </td>
                        <td>
                            <a href='mailto:customersupport@linz.govt.nz'>Contact Us</a>
                        </td>
                    </tr>
                </table>
            </div>
        </div>
    </body>
</html>
SPlanzer commented 6 years ago

I have added a retry in the case of a 504. The the layer still can not be attained this is logger out as an error and raised with the user

https://github.com/linz/lds-metadata-updater/blob/4020bf20e0fdc2b288321bfd265023756370abfb/metadata_updater/metadata_updater.py#L331-L341