search

danieldotnl / ha-multiscrape

Home Assistant custom component for scraping (html, xml or json) multiple values (from a single HTTP request) with a separate sensor/attribute for each value. Support for (login) form-submit functionality.
MIT License
301 stars 17 forks source link

Script is not activate? #452

Closed Feneck91 closed 5 days ago

Feneck91 commented 2 weeks ago

Version of the custom_component : 8.0.3

Configuration

The yaml is load from configuration.yaml with multiscrape: !include /config/YAML/automatic_tarifs_tempo.yaml

- name: Tarifs EDF Tempo
  resource: "https://particulier.edf.fr/fr/accueil/espace-client/selecteur-contrat.html?goto=%2Ffr%2Faccueil%2Fespace-client%2Fmes-contrats.html"
  scan_interval: 60
  log_response: true
  form_submit:
    submit_once: true
    resource: "https://espace-client.edf.fr/sso/XUI/#login/&realm=%2FINTERNET&goto=https%3A%2F%2Fespace-client.edf.fr%2Fsso%2Foauth2%2FINTERNET%2Fauthorize%3Fresponse_type%3Dcode%26email_hint%3DTHEMAILADRESS%2540gmail.com"
    select: "#openam-form-div > form"
    input:
      email: THEMAILADRESS%2540gmail.com
      passwordHid: "THEPASSWORD"
  sensor:
    - unique_id: puissance_souscrite
      name: Puissance souscrite
      select: "#power"
      value_template: "{{ value.replace('kVA', '') }}"

Describe the bug

The log seem indicate that the script is not running, please activate javascript into your browser. I'm not sur of the "resource" url..

In fact, the real url without the mail (redirect the url = https://particulier.edf.fr/fr/accueil/espace-client/selecteur-contrat.html?goto=%2Ffr%2Faccueil%2Fespace-client%2Fmes-contrats.html) into the url display a page where you must input the mail address then, after submit, input the password. Not sure this tool is able to login with 2 steps.

Debug log

2024-11-17 15:28:51.085 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Tarifs EDF Tempo # Deleting logging files from previous run
2024-11-17 15:28:51.089 DEBUG (MainThread) [custom_components.multiscrape.form] Tarifs EDF Tempo # Starting with form-submit
2024-11-17 15:28:51.090 DEBUG (MainThread) [custom_components.multiscrape.form] Tarifs EDF Tempo # Requesting page with form from: https://espace-client.edf.fr/sso/XUI/#login/&realm=%2FINTERNET&goto=https%3A%2F%2Fespace-client.edf.fr%2Fsso%2Foauth2%2FINTERNET%2Fauthorize%3Fresponse_type%3Dcode%26email_hint%3Dschateau%2540laposte.net
2024-11-17 15:28:51.090 DEBUG (MainThread) [custom_components.multiscrape.http] Tarifs EDF Tempo # Executing form_page-request with a GET to url: https://espace-client.edf.fr/sso/XUI/#login/&realm=%2FINTERNET&goto=https%3A%2F%2Fespace-client.edf.fr%2Fsso%2Foauth2%2FINTERNET%2Fauthorize%3Fresponse_type%3Dcode%26email_hint%3Dschateau%2540laposte.net with headers: {} and cookies: None.
2024-11-17 15:28:51.093 DEBUG (MainThread) [custom_components.multiscrape.http] Tarifs EDF Tempo # request_headers written to file: form_page_request_headers.txt
2024-11-17 15:28:51.161 DEBUG (MainThread) [custom_components.multiscrape.http] Tarifs EDF Tempo # Response status code received: 200
2024-11-17 15:28:51.167 DEBUG (MainThread) [custom_components.multiscrape.http] Tarifs EDF Tempo # response_headers written to file: form_page_response_headers.txt
2024-11-17 15:28:51.169 DEBUG (MainThread) [custom_components.multiscrape.http] Tarifs EDF Tempo # response_body written to file: form_page_response_body.txt
2024-11-17 15:28:51.170 DEBUG (MainThread) [custom_components.multiscrape.http] Tarifs EDF Tempo # response_cookies written to file: form_page_response_cookies.txt
2024-11-17 15:28:51.170 DEBUG (MainThread) [custom_components.multiscrape.form] Tarifs EDF Tempo # Parse page with form with BeautifulSoup parser lxml
2024-11-17 15:28:51.232 DEBUG (MainThread) [custom_components.multiscrape.form] Tarifs EDF Tempo # The page with the form parsed by BeautifulSoup has been written to file: form_page_soup.txt
2024-11-17 15:28:51.232 DEBUG (MainThread) [custom_components.multiscrape.form] Tarifs EDF Tempo # Try to find form with selector #openam-form-div > form
2024-11-17 15:28:51.234 ERROR (MainThread) [custom_components.multiscrape.coordinator] Tarifs EDF Tempo # Exception in form-submit feature. Will continue trying to scrape target page.
Could not find form
2024-11-17 15:28:51.234 DEBUG (MainThread) [custom_components.multiscrape.http] Tarifs EDF Tempo # Executing page-request with a GET to url: https://particulier.edf.fr/fr/accueil/espace-client/selecteur-contrat.html?goto=%2Ffr%2Faccueil%2Fespace-client%2Fmes-contrats.html with headers: {} and cookies: None.
2024-11-17 15:28:51.242 DEBUG (MainThread) [custom_components.multiscrape.http] Tarifs EDF Tempo # request_headers written to file: page_request_headers.txt
2024-11-17 15:28:51.629 DEBUG (MainThread) [custom_components.multiscrape.http] Tarifs EDF Tempo # Response status code received: 200
2024-11-17 15:28:51.638 DEBUG (MainThread) [custom_components.multiscrape.http] Tarifs EDF Tempo # response_headers written to file: page_response_headers.txt
2024-11-17 15:28:51.643 DEBUG (MainThread) [custom_components.multiscrape.http] Tarifs EDF Tempo # response_cookies written to file: page_response_cookies.txt
2024-11-17 15:28:51.646 DEBUG (MainThread) [custom_components.multiscrape.http] Tarifs EDF Tempo # response_body written to file: page_response_body.txt
2024-11-17 15:28:51.648 DEBUG (MainThread) [custom_components.multiscrape.scraper] Tarifs EDF Tempo # Loading the content in BeautifulSoup.
2024-11-17 15:28:51.677 DEBUG (MainThread) [custom_components.multiscrape.scraper] Tarifs EDF Tempo # page_soup written to file: page_soup.txt
2024-11-17 15:28:51.678 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Tarifs EDF Tempo # Data successfully refreshed. Sensors will now start scraping to update.
2024-11-17 15:28:51.678 DEBUG (MainThread) [custom_components.multiscrape.coordinator] Finished fetching multiscrape data in 0.594 seconds (success: True)
2024-11-17 15:28:51.679 DEBUG (MainThread) [custom_components.multiscrape.sensor] Tarifs EDF Tempo # Puissance souscrite # Start scraping to update sensor
2024-11-17 15:28:51.681 DEBUG (MainThread) [custom_components.multiscrape.scraper] Tarifs EDF Tempo # Puissance souscrite # Tag selected: None
2024-11-17 15:28:51.682 DEBUG (MainThread) [custom_components.multiscrape.form] Tarifs EDF Tempo # Exception occurred while scraping, will try to resubmit the form next interval.
2024-11-17 15:28:51.682 ERROR (MainThread) [custom_components.multiscrape.sensor] Tarifs EDF Tempo # Puissance souscrite # Unable to scrape data: Could not find a tag for given selector 
Consider using debug logging and log_response for further investigation.
2024-11-17 15:28:51.684 DEBUG (MainThread) [custom_components.multiscrape.sensor] Tarifs EDF Tempo # Puissance souscrite # On-error, set value to None
2024-11-17 15:28:51.685 DEBUG (MainThread) [custom_components.multiscrape.entity] Tarifs EDF Tempo # Puissance souscrite # Sensor updated and state written to HA

File form_page_response_body.txt :


The contents of this file are subject to the terms of the Common Development 
and Distribution License (the License). You may not use this file except in 
compliance with the License.
You can obtain a copy of the License at legal/CDDLv1.0.txt. See the License 
for the specific language governing permission and limitations under the 
License. When distributing Covered Software, include this CDDL Header Notice
 in each file and include the License file at legal/CDDLv1.0.txt. 
If applicable, add the following below the CDDL Header, with the fields
enclosed by brackets [] replaced by your own identifying information: 
"Portions Copyright [year] [name of copyright owner]".

Copyright 2018-2021 TOSIT
-->
<!DOCTYPE html>
<html lang="fr" style="overflow: auto;">

<head>
    <meta http-equiv="Content-Type" ; content="charset=utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="robots" content="index,follow">
    <meta name="description" content="Accéder à votre compte EDF pour gérer votre contrat d’électricité et/ou gaz, suivre votre consommation d’énergie ou télécharger votre facture EDF.">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <noscript>
        <link rel="stylesheet" type="text/css" href="styles/mainErrors.css?v=7.3.0">
        <link rel="stylesheet" type="text/css" href="styles/utilities/bootstrap-grid.css?=7.3.0">
        <link rel="stylesheet" type="text/css" href="styles/utilities/reset.css?=7.3.0">
    </noscript>
    <script src="libs/lodash-3.10.1-min.js"></script>
    <script type="text/javascript" src="js/mobile-detect.min.js?v=7.3.0"></script>
    <script type="text/javascript" src="libs/jquery-3.3.1.min.js?v=7.3.0"></script>
    <script src="libs/trackingTC.js?v=7.3.0"></script>
    <script src="libs/tracking2.js?v=7.3.0"></script>
    <title>Me connecter ou créer mon espace client EDF</title>

<script>(window.BOOMR_mq=window.BOOMR_mq||[]).push(["addVar",{"rua.upush":"false","rua.cpush":"false","rua.upre":"false","rua.cpre":"false","rua.uprl":"false","rua.cprl":"false","rua.cprf":"false","rua.trans":"","rua.cook":"false","rua.ims":"false","rua.ufprl":"false","rua.cfprl":"false","rua.isuxp":"false","rua.texp":"norulematch","rua.ceh":"false","rua.ueh":"false","rua.ieh.st":"0"}]);</script>
                              <script>!function(e){var n="https://s.go-mpulse.net/boomerang/";if("False"=="True")e.BOOMR_config=e.BOOMR_config||{},e.BOOMR_config.PageParams=e.BOOMR_config.PageParams||{},e.BOOMR_config.PageParams.pci=!0,n="https://s2.go-mpulse.net/boomerang/";if(window.BOOMR_API_key="GGFC8-WD672-UKFXN-XLZD4-G6LRQ",function(){function e(){if(!r){var e=document.createElement("script");e.id="boomr-scr-as",e.src=window.BOOMR.url,e.async=!0,o.appendChild(e),r=!0}}function t(e){r=!0;var n,t,a,i,d=document,O=window;if(window.BOOMR.snippetMethod=e?"if":"i",t=function(e,n){var t=d.createElement("script");t.id=n||"boomr-if-as",t.src=window.BOOMR.url,BOOMR_lstart=(new Date).getTime(),e=e||d.body,e.appendChild(t)},!window.addEventListener&&window.attachEvent&&navigator.userAgent.match(/MSIE [67]\./))return window.BOOMR.snippetMethod="s",void t(o,"boomr-async");a=document.createElement("IFRAME"),a.src="about:blank",a.title="",a.role="presentation",a.loading="eager",i=(a.frameElement||a).style,i.width=0,i.height=0,i.border=0,i.display="none",o.appendChild(a);try{O=a.contentWindow,d=O.document.open()}catch(_){n=document.domain,a.src="javascript:var d=document.open();d.domain='"+n+"';void 0;",O=a.contentWindow,d=O.document.open()}if(n)d._boomrl=function(){this.domain=n,t()},d.write("<bo"+"dy onload='document._boomrl();'>");else if(O._boomrl=function(){t()},O.addEventListener)O.addEventListener("load",O._boomrl,!1);else if(O.attachEvent)O.attachEvent("onload",O._boomrl);d.close()}function a(e){window.BOOMR_onload=e&&e.timeStamp||(new Date).getTime()}if(!window.BOOMR||!window.BOOMR.version&&!window.BOOMR.snippetExecuted){window.BOOMR=window.BOOMR||{},window.BOOMR.snippetStart=(new Date).getTime(),window.BOOMR.snippetExecuted=!0,window.BOOMR.snippetVersion=14,window.BOOMR.url=n+"GGFC8-WD672-UKFXN-XLZD4-G6LRQ";var i=document.currentScript||document.getElementsByTagName("script")[0],o=i.parentNode,r=!1,d=document.createElement("link");if(d.relList&&"function"==typeof d.relList.supports&&d.relList.supports("preload")&&"as"in d)window.BOOMR.snippetMethod="p",d.href=window.BOOMR.url,d.rel="preload",d.as="script",d.addEventListener("load",e),d.addEventListener("error",function(){t(!0)}),setTimeout(function(){if(!r)t(!0)},3e3),BOOMR_lstart=(new Date).getTime(),o.appendChild(d);else t(!1);if(window.addEventListener)window.addEventListener("load",a,!1);else if(window.attachEvent)window.attachEvent("onload",a)}}(),"".length>0)if(e&&"performance"in e&&e.performance&&"function"==typeof e.performance.setResourceTimingBufferSize)e.performance.setResourceTimingBufferSize();!function(){if(BOOMR=e.BOOMR||{},BOOMR.plugins=BOOMR.plugins||{},!BOOMR.plugins.AK){var n=""=="true"?1:0,t="",a="kjalsqqccqfcqzzz74hq-f-2d94f3904-clientnsv4-s.akamaihd.net",i="false"=="true"?2:1,o={"ak.v":"39","ak.cp":"635978","ak.ai":parseInt("416349",10),"ak.ol":"0","ak.cr":7,"ak.ipv":4,"ak.proto":"http/1.1","ak.rid":"c52581e","ak.r":29288,"ak.a2":n,"ak.m":"a","ak.n":"essl","ak.bpcip":"82.64.185.0","ak.cport":50522,"ak.gh":"2.16.165.135","ak.quicv":"","ak.tlsv":"tls1.3","ak.0rtt":"","ak.0rtt.ed":"","ak.csrc":"-","ak.acc":"reno","ak.t":"1731854095","ak.ak":"hOBiQwZUYzCg5VSAfCLimQ==AK3d6EAhX0ViN/VANMdcC9aOMsYufoA9iKPKpp/3VdxXsYprILPyhB6LlC+mPNt/az6wXToVZcbE3q884x2oUik30KdCIZVBX0oQZdj6JIbFr1VvBk0fFD/rRbpuiew7WrwftovyQwyLxSt4ytCViBje63/PzuRK04sxtkiSOWqhFhFmV1MwoiVm9j7E48P2uMt5NMTUTcLjVChkH1F9SXmTsgUnwIR+1it9XOXiJ64TNTg8WV8oQY4nG5LCt2g2tDwY8AOxTUMsSA7tPyAXq6V2H4oxUmQLe5QI5YEMSqy18xrKJ6+/WILzeJaeo8NRswbI5a31AqFiMGtOsbeNpUZ2r7hufvPnfWT2FKQDRrzBR98zQwFPCd9Nl4cuAtnSrql8vU9ZVQQhCcLKNHpkAF8N23o417UzLfzcn51zpYM=","ak.pv":"105","ak.dpoabenc":"","ak.tf":i};if(""!==t)o["ak.ruds"]=t;var r={i:!1,av:function(n){var t="http.initiator";if(n&&(!n[t]||"spa_hard"===n[t]))o["ak.feo"]=void 0!==e.aFeoApplied?1:0,BOOMR.addVar(o)},rv:function(){var e=["ak.bpcip","ak.cport","ak.cr","ak.csrc","ak.gh","ak.ipv","ak.m","ak.n","ak.ol","ak.proto","ak.quicv","ak.tlsv","ak.0rtt","ak.0rtt.ed","ak.r","ak.acc","ak.t","ak.tf"];BOOMR.removeVar(e)}};BOOMR.plugins.AK={akVars:o,akDNSPreFetchDomain:a,init:function(){if(!r.i){var e=BOOMR.subscribe;e("before_beacon",r.av,null,null),e("onbeacon",r.rv,null,null),r.i=!0}return this},is_complete:function(){return!0}}}}()}(window);</script></head>

<!--[if IE 9]>
    <body style="display:none" class="ie9">
    <![endif]-->
<!--[if (gt IE 9)|!(IE)]><!-->

<body style="overflow: auto;">
    <noscript>
        <div id="section">
            <img class="section-logo" src="img/zone1_desktop_default.svg" alt="">
            <h1 class="section-title">Je me connecte à mon espace client particulier</h1>
        </div>
        <div id="content">
            <div class="errors-pages js-document">
                <div class="errors-pages__id">
                </div>
                <section>
                    <main class="errors-pages__content">
                        <h1>
                            JavaScript requis
                        </h1>
                        <p>
                            JavaScript est d&eacute;sactiv&eacute; dans votre navigateur.
                        </p>
                        <p class="alert-information">
                            Pour vous connecter, veuillez activer JavaScript dans les param&egrave;tres de votre navigateur.
                        </p>
                    </main>
                </section>
            </div>
        </div>
    </noscript>
    <!--<![endif]-->
    <div id="messages" class="clearfix"></div>
    <div id="wrapper"></div>
    <div id="popup">
        <div id="popup-content" class="radious"></div>
    </div>
    <script type="text/javascript">
        var require = {
            urlArgs: "v=7.3.0",
            deps: ['main']
        };
        if (!String.prototype.startsWith) {
            String.prototype.startsWith = function(searchString, position) {
                position = position || 0;
                return this.substr(position, searchString.length) === searchString;
            };
        }

        var urlFull=window.location.href;
        var urlHash=window.location.hash;

        if (urlFull.indexOf('goto') < 0 && (urlHash.indexOf('login') > 0 || urlHash == "") && (urlFull.indexOf('entreprises-collectivites') > 0)) {
        // Récupérer hostname actuel   
        var hostname = window.location.hostname;
        // Vérifier l'hostname pour rediriger vers les URL appropriées
        switch (hostname) {
        case 'auth.entreprises-collectivites.edf.fr':  // Prod
            window.location.href = "https://entreprises-collectivites.edf.fr";
            break;
        case 'auth-rbb.entreprises-collectivites.edf.fr':  // RBB
            window.location.href = "https://rispi.entreprises-collectivites.edf.fr";
            break;
        case 'auth-r7b.entreprises-collectivites.edf.fr':  // R7B
            window.location.href = "https://mco.entreprises-collectivites.edf.fr";
            break;
        default:  // R7F ou autres
            window.location.href = "https://valintp.entreprises-collectivites.edf.fr";
            break;
        }
     }
    </script>
    <script type="text/javascript" src="js/common.js?v=7.3.0"></script>
    <script type="text/javascript" src="./atomx/components/atomx-input/js/atomx-input.js?v=7.3.0"></script>
    <script type="text/javascript" src="./atomx/components/atomx-select/js/atomx-select.js?v=7.3.0"></script>
    <script type="text/javascript" src="./atomx/components/atomx-modal/js/atomx-modal.js?v=7.3.0"></script>
    <script type="text/javascript" src="./atomx/components/atomx-modal/js/micromodal.min.js?v=7.3.0"></script>
    <script type="text/javascript" src="libs/requirejs-2.1.14-min.js?v=7.3.0"></script>
<script type="text/javascript" nonce="258a9e99030fdfead1f164628e32105c" src="/ZbdDWcthMttJnEwndw/5VukrtbzbhLJ3w/GQ41Sg5kMgI/Ki/coWU86ZmgB"></script></body>

</html>```
danieldotnl commented 5 days ago

Please ask support questions on the community forum. You might still be able to login, but you need to investigate the requests that the browser is doing in the developer tools.