search

reyemb / AISExplorer

This tool allows the identification and tracking of marine vessels by geolocation, MMSI, or via MarineTraffic URLs. It ensures user privacy via proxy support, and includes fallback strategies for optimal performance.
27 stars 8 forks source link

Scraper not working properly #2

Closed itec78 closed 1 year ago

itec78 commented 1 year ago

Hi. I think that something is broken. I'm receiving bad answers for all the queries I tried.

Using first code in example:

from aisexplorer.AIS import AIS
AIS().get_location(211281610)

get this:

Response code: 403 - headers: {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Vessel-Image': '005bf958a6548a79c6d3a42eba493e339624'} <!DOCTYPE html>
<html lang="en-US">
<head>
    <title>Just a moment...</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=Edge">
    <meta name="robots" content="noindex,nofollow">
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <link href="/cdn-cgi/styles/challenges.css" rel="stylesheet">

</head>
<body class="no-js">
    <div class="main-wrapper" role="main">
    <div class="main-content">
        <noscript>
            <div id="challenge-error-title">
                <div class="h2">
                    <span class="icon-wrapper">
                        <div class="heading-icon warning-icon"></div>
                    </span>
                    <span id="challenge-error-text">
                        Enable JavaScript and cookies to continue
                    </span>
                </div>
            </div>
        </noscript>
        <div id="trk_jschal_js" style="display:none;background-image:url('/cdn-cgi/images/trace/managed/nojs/transparent.gif?ray=7c3b2a8ab8e60e83')"></div>
        <form id="challenge-form" action="/en/reports?asset_type=vessels&amp;columns=time_of_latest_position,flag,shipname,photo,recognized_next_port,reported_eta,reported_destination,current_port,imo,mmsi,ship_type,show_on_live_map,area,area_local,lat_of_latest_position,lon_of_latest_position,fleet,status,eni,speed,course,draught,navigational_status,year_of_build,length,width,dwt,current_port_unlocode,current_port_country,callsign&amp;mmsi%7Ceq%7Cmmsi=211281610&amp;__cf_chl_f_tk=5lumYa8u0Q0vGXEyl7aDdqDED60HcE1cQby3PRkjsu4-1683480367-0-gaNycGzND3s" method="POST" enctype="application/x-www-form-urlencoded">
            <input type="hidden" name="md" value="T0A9Q_oWWTNJ90qfXxdKt7Kwd5GWi2HMmEERwiWf.YE-1683480367-0-AcanNC-UP989ezPBKwc5slRCauGpYvNedJ-BUPQzWF062vpPdVQsTT1-tJJ8Osqp8__2cxZPliy1Ikb-_r84MeU4foBC4a3kX6aEddMtr1g-4bmzsJcRgknGOFa4QkwSNi5D5xov-Emlh7b5lRRPrOAUue3thMEyHvEbJhMG2fSif1xQ3U24QM2uJR94kshPObC7tPE_GYY0wO4F4OjwT8Ej2-dRmxtyGweEo9TvqiQFUB7ocJOqE_GL1_wCXJ3aBrLurBkpgAOXl5368HK_twvHbRK0G3fUbQqfVWDQ8Yy3SuZmUV2T00wgK1nocpJMsSNiO2FbeORCtyNXQY4Lkr0V13PhjlFdKJjD0kfLXawIcNwLpBxF6HqBp2d5RP73AtCZ7pAVwzAGIQL7Dxvc5SJ_wxkCcmsH3hIT_wilOAZ234lntIZNxQGmlTSgs-D6OJf-OUibuONhBtE15byTA1c_WS4xbmQom8Tz0HFzNb14WmJPF9q8ZqUPTNhPVxb_7p6yDSVAxaUdZhnW58PvJQQvE09QYnOCBoaJh8uYKBfaCuJO9ckW8PRPrBfyOnULTxuGb2y234fGfIQH3HORYoeC4u8rIQ5FUuW9pJilmgJBIt-7gtoKS4DC72x456nAaqAgDKN7HKvzmQNVPBwhI0QghVelsRO_JH1q79lrGTLITGPKNOo2S7Yr4Zl19XN_R8WE5fX1hGMmtJhFTytSHyNcAu9apqfQ7VSxxZl2LVH_Uoc3Sb3Mi2vamUozqiJ30KO5dz-zR1UqyirSxDwXNZCXVsB1mLQftQKoa-XSln8z3lGLd6Yi5hGDjDuFFU9Q-VDF1JZq8a3rKDZse8ch2kqct7hxk27UTmR2178V5yHjpcd9Mpoz_DVGu5LD2tq0AiMGe-fsEpQOLR2FSNLKiAiOxMsSLDO5C2btgbjNfVojohliYMHlTR2SLE-vGOFFKqU2xtm8S61IZgYcsiyAyagVAetUg4X5-pCiNLN9BwQF8ocCnfwUg9xlNFPnpb-uYMZXZM58QinR5oQLMEwTQD31MylSCVasWfKlpHKls0mG3g1nBPDOiAzmaTBWXTCrDuBVSAcYv0LTL5HI4-Dk0exkJjGsDFVnzXAF3XuVdfT6-Nn4lANkT6UqjbjezIJ62Gwsh14VphvD5Ii2QVcbqeRNpJuokr4x0GK_HSPUg6Q1C1OLnf3LRbVh4JJUe55H0EH5zFBTzyUfSFOGM23OisSvUWH1wiYVgYH0qVLVKrXLe_6srHU5qCYxc6vILriYSo8EWfxIpvjS6C_9Zcvha2icj5_5rFfEdV4oT8jdlIEkdTIbZ1udYi6ikclh3pOQPnuTaejmiH37Fk7XZq48fpFqzNunT2p4emw30h_Je-TKJ37JGtmG31WlIagNI3Agyos9e-a2okUFn5KHEomxhwGX-iuMs0Pf-d-80pmtlZSVox__5odxbPkPraIlv3jQH7ESTVJUAc2dtSPR59SAEpHgG12H0_whplmpFX6y0YLAAE_iKipswwMG-5ld4tCxK1dL1urDTujt4ohvRQscgVJnFvU8CkYVHaT3-y681B0ZA50u3nLfe5HHzYsXnbyQlRnslUViA7f7e3-AoMoiHdn_Kqs0tu6O1gK94KmqDG1nlikLle7zUgOUyEG7FiU5K_HJ_2jYGjOnbq4_VIomU6xpxiejiwwbbMG48HhK4U_G2Apz7_Fyhi2xzbTiTSpr4kHoS_1gAILBs637Fd4Ym1WBfjMKv7dW1fN8iuOYmreo07v54qvvHwJmzbaVm9DNSJM3Ef0TmJCUpFAqONt2KgHm1iPXxoiK0I6ZSgLA_C-iwRpU_q6txXG_vvBc9csAEgIvlH2AvDLDhmLxkJXXEYuuUzB5jwpcDcRimXuQfedNSWVBpAEE1eExRP7mPk7jknU9A24sX1ptbv_ztoti1Lztxgaj8Q9S9aq0oONoAjttkwrEF3gUznLlTURlUc2TovAe5WgwLQfl3nCkXkyYjpmKKaqJwLQos3W95nj0drk_QE664jRvtOvqgGbtNTkm1Ng9YMUeVrJpMgNqR4MZE7iFzBW1tR4SRa0CatM4FVUoKDp5qyU3s0hH7xenjFveyk3IP9Y1xoay1_yLY3q5NF4mmkUET-_ujvwZyXbvB4ssWoPGSlkDjsK4uWO0JYkSMOLQTj_x6UJvvEp6s7TqSmU7OEMfnNxnqDAQ1Xy5vnxM0hnbNSUaxa8LaD96beFFcSqxCMhY-4Nee_oLp17sUwy7UJMP-OBGaIzH3fgGvhovrSIECKcPRooaScHTuvFYIexxYcv2_dK6UQrzbiGTUmjGf2jqhpcZ7oL2LN3AG4lrlIFr8t18lCikLuHipVLZhcTY4T-5bBSxIq0LLPjFfYJAIly-GZCQAmiuI8AU16FiRYTZIV3dGtXZGHG_xvIIV4QY-LnBKC2zK3UqoJEwVotDjYjtahQ0U16KURQXfAAvCQm4Hztw4emF94yJp4TgumCN_V2PjK9tbo3N-a6IVghUogTI4saqAPIakxt6FCP26EZNnt2islUEGdyjnLl7DF3mV23q_wTBu8zSPdRTpP_88VB2z1RhRsy6RBr8riygz3C4kI1E2W0E3K4leFpGLIBPP_1PimAdZ0ZRpQVSAoAB2lgXe-zK2hLuGUlWwslmeBioVIFsZdO3ougya14PX94YcypCJX0yKsqA2AjxdAUNzJe1oA2NbjVLkH7v2lAThkPp_-u5nKuB3R4VYkHPZT4aVYIq628Nno8thHEmsZhuZ9mOX7WoYJ82V1HubS2dBmJX6bSm8k6pU4eFs2djiefMDEHdydxxXNN9SnrL3Nsg-w5577lQxYY4Q1GLTa1pyY1aeDrqznR8lOMl0jaKjY_3mDkGSBUVNHBIDJBFnZ_RfaWBvjew-QkfBAbFv03rmXkeeAm5zziBGT8EZat0wMKLXiIpIXCFqi7WHguNMjMpNm7DAGA59PKkfg_Me5wHt5dNLfO-fto45YEAeNKxqiA8T6EhMUbO7MHEsWv9boRVpEmsebe4JHo1FnymUs9siIuHm851V55POXZzLcwZLGnzlfLmqcqZvOruPEpYPOjVa5w8atzzZRnIeGDcRNn9_RDy5sPFo6art-1_FrN46xrXdZSLW9KOguNeV4DAJmjcDqS3rqeLGnXbVpzzbIroRTH6v_mhkWeznPjU0EkImXJSSluMXgRwxQsLxtTcvkW0pfmhkMxTWsmqqPv7tl-sAfaNb7Vgb0-i96NUIrZ3NJLZ4uM3s4iH8vVRZ5VSh1U8X-vt18LPPPh4hB9NkoPM1bAGR1xQdSX8gemRn6H0peRv3R0lC8iBf7DmTjJUF23pKd22mtwW32YMbp7Im9lSVIhgUzv-v32-oDOtdhaTEUOnbNb4mhfYn2YyavmbhNc2o8GQ9XrAgAmO2nFdpEzSmyDOBaEexJc2JWGMqt_c1QZfudZTV7KtVZ6hBXwAx8nLXMQw52mBi5ynlcFl8FkAzNPqt_Y-ZGBg6FDR9GYJMZsk5aXkJvwE8jNxP8jcG98KsJtNMr7Y8p_7kxgtvolyXGbH7jrNg3j5mdXP--ToNgXfO1Uxhr3Me58HgR7miXw-HzbIAH_QOsI8KDAnQ4gYFHW5uj5narzO9vHqyVIegBe763BSQlx-5fFN9BkaNm5JzRVl-Rp1iTeMe6cpAe1NQJvdDMFKwTO_TZ3BeGq8PDZhMZxfRxhpjc8kEmDO-6K0xHzA5olzcYgGmH55r67ToKqcd7apj42Oyd4f0_M36ipP3-nP2-3kdP3vtJi-8PXmkGSUcPHvkvjKvyjse4WtZvVxASm2jOeyfAhrnv0K_XLb8ksL7u5tu9aCHTX3QW2M-i85pGt97cq7wOO3OnmFZ5BTgB9h4dtM0IpsC52p0A">
        </form>
    </div>
</div>
<script>
    (function(){
        window._cf_chl_opt={
            cvId: '2',
            cZone: 'www.marinetraffic.com',
            cType: 'managed',
            cNounce: '80480',
            cRay: '7c3b2a8ab8e60e83',
            cHash: 'fbedeeeda5c255e',
            cUPMDTk: "\/en\/reports?asset_type=vessels&columns=time_of_latest_position,flag,shipname,photo,recognized_next_port,reported_eta,reported_destination,current_port,imo,mmsi,ship_type,show_on_live_map,area,area_local,lat_of_latest_position,lon_of_latest_position,fleet,status,eni,speed,course,draught,navigational_status,year_of_build,length,width,dwt,current_port_unlocode,current_port_country,callsign&mmsi%7Ceq%7Cmmsi=211281610&__cf_chl_tk=5lumYa8u0Q0vGXEyl7aDdqDED60HcE1cQby3PRkjsu4-1683480367-0-gaNycGzND3s",
            cFPWv: 'g',
            cTTimeMs: '1000',
            cMTimeMs: '0',
            cTplV: 5,
            cTplB: 'cf',
            cK: "",
            cRq: {
                ru: 'aHR0cHM6Ly93d3cubWFyaW5ldHJhZmZpYy5jb20vZW4vcmVwb3J0cz9hc3NldF90eXBlPXZlc3NlbHMmY29sdW1ucz10aW1lX29mX2xhdGVzdF9wb3NpdGlvbixmbGFnLHNoaXBuYW1lLHBob3RvLHJlY29nbml6ZWRfbmV4dF9wb3J0LHJlcG9ydGVkX2V0YSxyZXBvcnRlZF9kZXN0aW5hdGlvbixjdXJyZW50X3BvcnQsaW1vLG1tc2ksc2hpcF90eXBlLHNob3dfb25fbGl2ZV9tYXAsYXJlYSxhcmVhX2xvY2FsLGxhdF9vZl9sYXRlc3RfcG9zaXRpb24sbG9uX29mX2xhdGVzdF9wb3NpdGlvbixmbGVldCxzdGF0dXMsZW5pLHNwZWVkLGNvdXJzZSxkcmF1Z2h0LG5hdmlnYXRpb25hbF9zdGF0dXMseWVhcl9vZl9idWlsZCxsZW5ndGgsd2lkdGgsZHd0LGN1cnJlbnRfcG9ydF91bmxvY29kZSxjdXJyZW50X3BvcnRfY291bnRyeSxjYWxsc2lnbiZtbXNpJTdDZXElN0NtbXNpPTIxMTI4MTYxMA==',
                ra: 'TW96aWxsYS81LjAgKFdpbmRvd3MgTlQgMTAuMDsgV2luNjQ7IHg2NCkgQXBwbGVXZWJLaXQvNTM3LjM2IChLSFRNTCwgbGlrZSBHZWNrbykgQ2hyb21lLzEwOC4wLjAuMCBTYWZhcmkvNTM3LjM2',
                rm: 'R0VU',
                d: '9UmkfU54/yHlLK2blqdZT0IBQgM/eh1L6qHHbLgbN6GLx8JhKbjmQtJDI7wKYHzJSn6VakJMB4vZ5DpcdSF7mUztNmomx0LAL7vgHzjx+vtHVbzfulmOctt8QAYLgBVhF1Iot7+Qkst/9bRSMtQqDFuNNjjJvy7mVx34K92OiWq83ANVOgXyQq0yGhKPX4GjBG4JS1aTiU0O1PZJr8A9DTsuYEHRhfBK6kwypmrQoNIFFBDBhL1O9vDtMUiJ3BxAuG4mLgzxWvkcELdi2asnN0c1TT6GNRSh+tfgp3fGTTO75NL7ERKW3tEoA7wqeaG6LqV+zoTs4qTAAfNE9mY6l0v/B780P8gdoxm4OSNv31sAZMiwS9iCh2u9aS0QmpGX90qlLO5zXD1Bd/DYS0nwilHUuuZjCAcPisM6NveXA3dPEEBqbaa5L6kLwQLfI7kgSqv/F9X98KWR4aBmvEoLza0zC33b3MK4bipJ0fgbEjzGPEcsyxK2q7jG/zmutz5CUoelwxxvdBuX3ebd9IkaWkJ5zzbn5hdCtEoK8niZO9rLD4svIy3Ndi3ekDf7gTiSwDEReVVL3s/jpxC1E4NlaKozoBVNn7ekF/cKizTZ145H6VDsW3xHqq3IYrRLv7nR',
                t: 'MTY4MzQ4MDM2Ny44MDkwMDA=',
                m: 'jW3sPvu77ZpogKHxA0t0C7d8rM2XLHjjwSXEkSbWy0o=',
                i1: 'uQt3aWHaMbHxXSVN54HIPg==',
                i2: 'pDEbQ9i8KJhOeILugKX5Uw==',
                zh: '9ODJ9wUnoPjm/IAh4FoxBVEPcuhbtkHqgDVY7Mg1mFw=',
                uh: 'JkAkY486s8b7quQvdaIQRGNNNpmu+ejtUZNcz6o7wFM=',
                hh: 'qi1pc4iF5r4NNN+eu2J/M9aMOtWqOev4stD1qgOIFxo=',
            }
        };
        var trkjs = document.createElement('img');
        trkjs.setAttribute('src', '/cdn-cgi/images/trace/managed/js/transparent.gif?ray=7c3b2a8ab8e60e83');
        trkjs.setAttribute('alt', '');
        trkjs.setAttribute('style', 'display: none');
        document.body.appendChild(trkjs);
        var cpo = document.createElement('script');
        cpo.src = '/cdn-cgi/challenge-platform/h/g/orchestrate/managed/v1?ray=7c3b2a8ab8e60e83';
        window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;
        window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, location.href.length - window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;
        if (window.history && window.history.replaceState) {
            var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;
            history.replaceState(null, null, "\/en\/reports?asset_type=vessels&columns=time_of_latest_position,flag,shipname,photo,recognized_next_port,reported_eta,reported_destination,current_port,imo,mmsi,ship_type,show_on_live_map,area,area_local,lat_of_latest_position,lon_of_latest_position,fleet,status,eni,speed,course,draught,navigational_status,year_of_build,length,width,dwt,current_port_unlocode,current_port_country,callsign&mmsi%7Ceq%7Cmmsi=211281610&__cf_chl_rt_tk=5lumYa8u0Q0vGXEyl7aDdqDED60HcE1cQby3PRkjsu4-1683480367-0-gaNycGzND3s" + window._cf_chl_opt.cOgUHash);
            cpo.onload = function() {
                history.replaceState(null, null, ogU);
            };
        }
        document.getElementsByTagName('head')[0].appendChild(cpo);
    }());
</script>

</body>
</html>
 - referer_url: https://www.marinetraffic.com/en/data/?asset_type=vessels&columns=time_of_latest_position,flag,shipname,photo,recognized_next_port,reported_eta,reported_destination,current_port,imo,mmsi,ship_type,show_on_live_map,area,area_local,lat_of_latest_position,lon_of_latest_position,fleet,status,eni,speed,course,draught,navigational_status,year_of_build,length,width,dwt,current_port_unlocode,current_port_country,callsign&mmsi|eq|mmsi=211281610 - request_url: https://www.marinetraffic.com/en/reports?asset_type=vessels&columns=time_of_latest_position,flag,shipname,photo,recognized_next_port,reported_eta,reported_destination,current_port,imo,mmsi,ship_type,show_on_live_map,area,area_local,lat_of_latest_position,lon_of_latest_position,fleet,status,eni,speed,course,draught,navigational_status,year_of_build,length,width,dwt,current_port_unlocode,current_port_country,callsign&mmsi|eq|mmsi=211281610

What's wrong? Thanks

reyemb commented 1 year ago

I apologize for the delayed response. I have investigated the issue you're experiencing, but I'm unable to replicate the error on my end.

Executing these two lines of code:

from aisexplorer.AIS import AIS
AIS().get_location(211281610)

returns the vessel data for me without any 403 errors. Typically, a 403 status code indicates that the client does not have the necessary permissions for the requested resource. It's a HTTP response status code that means the server understood the request, but it refuses to authorize it. However, as there is no login functionality implemented at this stage, I'm unsure how this error is occurring.

To assist me further in diagnosing the issue, could you please provide me with the version of the libraries you are using? You can do this by running the following command:

pip freeze

Thank you for your patience.

itec78 commented 1 year ago

Now it's working fine. It stopped working on 2023-05-04 13:54 CEST and it started working again on 2023-07-05 21:13 CEST I don't know what happened in that period, probably something has changed on server side. Thanks a lot.