ianvanhoven commented 3 years ago

Starting ~2d ago (17-NOV), started getting these Personal Capital API login/auth errors...

% ./wx.ad.py ro Traceback (most recent call last): File "./wx.ad.py", line 60, in <module> pc.login( pc_cred_user , pc_cred_pass ) ; File "/usr/local/lib/python3.4/site-packages/personalcapital/personalcapital.py", line 44, in login raise Exception() Exception

...on code from this library that has run flawlessly for well over a year...

34 def login(self, username, password): 35 initial_csrf = self.__get_csrf_from_home_page(base_url) 36 csrf, auth_level = self.__identify_user(username, initial_csrf) 37 38 if csrf and auth_level: 39 self.__csrf = csrf 40 if auth_level != AuthLevelEnum.USER_REMEMBERED: 41 raise RequireTwoFactorException() 42 self.__authenticate_password(password) 43 else: 44 raise Exception()

Is anyone else experiencing this and/or have a fix/workaround? Tried disabling 2FA, no dice. Have logged a support case with PC but not holding my breath.

Thanks! Ian

coding4z commented 3 years ago

+1 with the exact same errors starting on 11/17/2021

ianvanhoven commented 3 years ago

Thx for verifying @khite1983 ... good in a way to know it's not specific/limited to me. Maybe chuck a Support case in as well.

(Unfortunately they didn't give me a case # for you to reference ... just an email thread, where I commented just now about your confirmation that this is broken.)

I'm dusting off my Plaid credentials & exploring that as an alternative if PC Support doesn't engage.

bhpetersheim commented 3 years ago

Same issue here as well.

UnusualPi commented 3 years ago

Looks like update in cloudflare, a small homepage structure change, and API payloads. My solution here (new dependency added, cloudscraper): pull request #21

bpassini12 commented 3 years ago

same issue.

ianvanhoven commented 3 years ago

Dumb question: how do I incorporate your changes before your PR is approved?

On Sat, Nov 20, 2021 at 1:36 PM UnusualPi @.***> wrote:

Looks like update in cloudflare, a small homepage structure change, and API payloads. My solution here (new dependency added, cloudscraper https://github.com/VeNoMouS/cloudscraper) #21 https://github.com/haochi/personalcapital/pull/21

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/haochi/personalcapital/issues/20#issuecomment-974714765, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEWS5DRVP5DMAWEPA5SYXVTUNAIE7ANCNFSM5IMRH7SQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

UnusualPi commented 3 years ago

@ianvanhoven it really depends on your implementation, but you'll definitely have to install cloudscraper. For the code changes in the personalcapital.py file, you could either edit directly in your python site-packages directory or you could just download the file, include in your script directory, and make the modifications on that file. I'd recommend the latter, when you use import in python it looks for files in the same directory before looking in site-packages. Hope that helps!

willyiwei commented 3 years ago

Thanks, @UnusualPi. I applied your fix and it resolved the login error issue on my end. I had to upgrade a couple of required package versions to make it fully work though. Thanks again for the solution!

banool commented 3 years ago

Was about to come here and make a similar pull request! Thanks for making this. For those coming along trying to fix this for themselves, run the following (assuming you're using requirements.txt):

pip uninstall personalcapital -y
pip install git+git://github.com/UnusualPi/personalcapital.git@cd0c88f78912c928ad8fafcce05864a43d914d54
pip freeze > requirements.txt
echo 'git+git://github.com/UnusualPi/personalcapital.git@cd0c88f78912c928ad8fafcce05864a43d914d54' >> requirements.txt

Note that there are breaking changes, such as authenticate_password requiring username now, so make sure to update your use of the library if you switch over.

ianvanhoven commented 3 years ago

Thanks @UnusualPi ... still at a loss on this :( Any additional suggestion/guidance appreciated.

I made your changes directly in...

/usr/local/lib/python3.4/site-packages/personalcapital/personalcapital.py

...re-confirmed I have cloudscraper...

% sudo pip install cloudscraper Requirement already satisfied: cloudscraper in /usr/local/lib/python3.4/site-packages (1.2.58) Requirement already satisfied: pyparsing>=2.4.7 in /usr/local/lib/python3.4/site-packages (from cloudscraper) (2.4.7) Requirement already satisfied: requests>=2.9.2 in /usr/local/lib/python3.4/site-packages (from cloudscraper) (2.20.0) Requirement already satisfied: requests-toolbelt>=0.9.1 in /usr/local/lib/python3.4/site-packages (from cloudscraper) (0.9.1) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.4/site-packages (from requests>=2.9.2->cloudscraper) (3.0.4) Requirement already satisfied: idna<2.8,>=2.5 in /usr/local/lib/python3.4/site-packages (from requests>=2.9.2->cloudscraper) (2.7) Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.4/site-packages (from requests>=2.9.2->cloudscraper) (1.24.3) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.4/site-packages (from requests>=2.9.2->cloudscraper) (2019.9.11)

...but getting the following error when executing my script...

% tail -0f log/2021/wx.log.202112.ad.err [Wed Dec 1 09:29:50 2021] ERROR: Package import failure; exiting... [Wed Dec 1 09:29:50 2021] ERROR: invalid syntax (init.py, line 202) ^C

...with init.py containing the following...

% cat init.py from .personalcapital import PersonalCapital, RequireTwoFactorException, TwoFactorVerificationModeEnum

...yet seeing the following (seemingly good) in STDOUT when running script w/ verbose logging...

% python -v wx.ad.py [ ... ]

code object from

/usr/local/lib/python3.4/site-packages/personalcapital/init.py

created

'/usr/local/lib/python3.4/site-packages/personalcapital/pycache/init.cpython-34.pyc'

wrote

'/usr/local/lib/python3.4/site-packages/personalcapital/pycache/init.cpython-34.pyc'

code object from

/usr/local/lib/python3.4/site-packages/personalcapital/personalcapital.py

created

'/usr/local/lib/python3.4/site-packages/personalcapital/pycache/personalcapital.cpython-34.pyc'

wrote

'/usr/local/lib/python3.4/site-packages/personalcapital/pycache/personalcapital.cpython-34.pyc'

destroy cloudscraper

destroy personalcapital.personalcapital

destroy personalcapital

[ ... ]

On Sun, Nov 21, 2021 at 7:14 AM UnusualPi @.***> wrote:

@ianvanhoven https://github.com/ianvanhoven it really depends on your implementation, but you'll definitely have to install cloudscraper. For the code changes in the personalcapital.py file, you could either edit directly in your python site-packages directory https://stackoverflow.com/questions/122327/how-do-i-find-the-location-of-my-python-site-packages-directory or you could just download the file, include in your script directory, and make the modifications on that file https://csatlas.com/python-import-file-module/. I'd recommend the latter, when you use import in python it looks for files in the same directory before looking in site-packages. Hope that helps!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/haochi/personalcapital/issues/20#issuecomment-974834963, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEWS5DXBW63BU7SSFVSLC7LUNEED7ANCNFSM5IMRH7SQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

ianvanhoven commented 3 years ago

LOL ... so took a wild guess that my old OpenBSD box where I host this script is just in some janky state -- and so I ported the script to a very recently-created AWS EC2 instance.

Good news: no more pkg import errors :)

Bad news: cloudscraper.exceptions.CloudflareChallengeError: Detected a Cloudflare version 2 Captcha challenge, This feature is not available in the opensource (free) version.

Is anyone else seeing this?

On Wed, Dec 1, 2021 at 10:05 AM Ian Van Hoven @.***> wrote:

Thanks @UnusualPi ... still at a loss on this :( Any additional suggestion/guidance appreciated.

I made your changes directly in...

/usr/local/lib/python3.4/site-packages/personalcapital/personalcapital.py

...re-confirmed I have cloudscraper...

% sudo pip install cloudscraper Requirement already satisfied: cloudscraper in /usr/local/lib/python3.4/site-packages (1.2.58) Requirement already satisfied: pyparsing>=2.4.7 in /usr/local/lib/python3.4/site-packages (from cloudscraper) (2.4.7) Requirement already satisfied: requests>=2.9.2 in /usr/local/lib/python3.4/site-packages (from cloudscraper) (2.20.0) Requirement already satisfied: requests-toolbelt>=0.9.1 in /usr/local/lib/python3.4/site-packages (from cloudscraper) (0.9.1) Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.4/site-packages (from requests>=2.9.2->cloudscraper) (3.0.4) Requirement already satisfied: idna<2.8,>=2.5 in /usr/local/lib/python3.4/site-packages (from requests>=2.9.2->cloudscraper) (2.7) Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.4/site-packages (from requests>=2.9.2->cloudscraper) (1.24.3) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.4/site-packages (from requests>=2.9.2->cloudscraper) (2019.9.11)

...but getting the following error when executing my script...

% tail -0f log/2021/wx.log.202112.ad.err [Wed Dec 1 09:29:50 2021] ERROR: Package import failure; exiting... [Wed Dec 1 09:29:50 2021] ERROR: invalid syntax (init.py, line 202) ^C

...with init.py containing the following...

% cat init.py from .personalcapital import PersonalCapital, RequireTwoFactorException, TwoFactorVerificationModeEnum

...yet seeing the following (seemingly good) in STDOUT when running script w/ verbose logging...

% python -v wx.ad.py [ ... ]

code object from

/usr/local/lib/python3.4/site-packages/personalcapital/init.py

created

'/usr/local/lib/python3.4/site-packages/personalcapital/pycache/init.cpython-34.pyc'

wrote

'/usr/local/lib/python3.4/site-packages/personalcapital/pycache/init.cpython-34.pyc'

code object from

/usr/local/lib/python3.4/site-packages/personalcapital/personalcapital.py

created

'/usr/local/lib/python3.4/site-packages/personalcapital/pycache/personalcapital.cpython-34.pyc'

wrote

'/usr/local/lib/python3.4/site-packages/personalcapital/pycache/personalcapital.cpython-34.pyc'

destroy cloudscraper

destroy personalcapital.personalcapital

destroy personalcapital

[ ... ]

On Sun, Nov 21, 2021 at 7:14 AM UnusualPi @.***> wrote:

@ianvanhoven https://github.com/ianvanhoven it really depends on your implementation, but you'll definitely have to install cloudscraper. For the code changes in the personalcapital.py file, you could either edit directly in your python site-packages directory https://stackoverflow.com/questions/122327/how-do-i-find-the-location-of-my-python-site-packages-directory or you could just download the file, include in your script directory, and make the modifications on that file https://csatlas.com/python-import-file-module/. I'd recommend the latter, when you use import in python it looks for files in the same directory before looking in site-packages. Hope that helps!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/haochi/personalcapital/issues/20#issuecomment-974834963, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEWS5DXBW63BU7SSFVSLC7LUNEED7ANCNFSM5IMRH7SQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

ecoen66 commented 2 years ago

I wonder if @haochi would be willing to assign ownership of this repo to another dev so that we could add this fix?

elstevega commented 2 years ago

Ok, I'm super close to getting this working...Getting the prompt for the verification code from Personal Capital, but when I click the 'Verify' button am getting the response: Failed to call service configurator/configure.authenticate_password() missing 1 required positional argument: 'password'

Any thoughts?

thanks

EDIT: It's line 74 of sensor.py

result = pc.authenticate_password(config.get(CONF_PASSWORD))

deusxanima commented 2 years ago

@elstevega You'll have to update the file you're running the main scripts in to include the additional parameter. UnusualPi's fix added an additional parameter that gets passed in and which is expected by the PersonalCapital parent class in the personalcapital module.

In your case above, you're passing in the password that you get with (config.get(CONF_PASSWORD) but because it's the only thing being passed in and not being passed in explicitly the parent class is treating is as the first parameter (username) and saying you're missing the password.

To fix it you can modify your authenticate_password method to pass in the email as well. I'm not sure what your actual script file looks like but you should be able to do something like result = pc.authenticate_password(config.get(CONF_EMAIL),config.get(CONF_PASSWORD)) (assuming you're able to reference both your email and password with config.get).

elstevega commented 2 years ago

@Aharic - thanks for the info - that did the trick. (I was close...) :)

elstevega commented 2 years ago

Argh, I spoke too soon. While the initial SMS validation piece now works (thanks @Aharic ), when I reboot the HA server, it loses the connection info and kicks back the below in the log file:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 249, in _async_setup_platform
    await asyncio.shield(task)
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/personalcapital/sensor.py", line 130, in setup_platform
    pc.login(config.get(CONF_EMAIL), config.get(CONF_PASSWORD))
  File "/usr/local/lib/python3.9/site-packages/personalcapital/personalcapital.py", line 44, in login
    raise Exception()
Exception

That bit of code is the update from UnusualPi...code block below - any thoughts? Thanks so much in advance!!

    def login(self, username, password):
        initial_csrf = self.__get_csrf_from_home_page(base_url)
        csrf, auth_level = self.__identify_user(username, initial_csrf)

        if csrf and auth_level:
            self.__csrf = csrf
            if auth_level != AuthLevelEnum.USER_REMEMBERED:
                raise RequireTwoFactorException()
            result = self.__authenticate_password(username, password).json()
            if getSpHeaderValue(result, SUCCESS_KEY) == False:
                raise LoginFailedException(getErrorValue(result))
        else:
            raise LoginFailedException()

willyiwei commented 2 years ago

@elstevega , I might be overlooking something, but judging by the Traceback infomation you pasted above, it seems like your code is still using the original version of the personalcapital.py from pip install, because the file path showed as:

File "/usr/local/lib/python3.9/site-packages/personalcapital/personalcapital.py"

That would still cause the issue that we are tracking here in this issue #20. Do you mind to double check your project folder and your python path. You may want to make sure your code is using the UnusualPi's updated personalcapital.py

elstevega commented 2 years ago

@elstevega , I might be overlooking something, but judging by the Traceback infomation you pasted above, it seems like your code is still using the original version of the personalcapital.py from pip install, because the file path showed as:

File "/usr/local/lib/python3.9/site-packages/personalcapital/personalcapital.py"

That would still cause the issue that we are tracking here in this issue #20. Do you mind to double check your project folder and your python path. You may want to make sure your code is using the UnusualPi's updated personalcapital.py

No problem & thanks for your response. Uninstalled/re-installed everything just to make sure. I cleared out the session on the personal capital website & re-authenticated. gonna reboot the HA server tomorrow & see what happens

willyiwei commented 2 years ago

Sounds good, @elstevega. And I had a brief glance of your other reply in my email inbox and from there I got some hints about what your issue might be.

When you get a chance to check it again, you may want to pay attention to the two different python lib paths below:

/usr/lib/python3.9/site-packages/personalcapital.py

v.s.

/usr/local/lib/python3.9/site-packages/personalcapital.py

By default, I think pip install the packages into /user/loca/lib/python3.9/....

Make sure you replace this file in the correct lib path. I use virtualenv so my lib file is inside my venv/ folder instead of global /usr/local/lib

You can use this command line to check the information about your pip installed personalcapital package:

$ pip show PersonalCapital

Then it will give you a Location to your personalcapital package installation on your host.

Hope it helps.

elstevega commented 2 years ago

Thanks @willyiwei - looks like I'm surviving reboots & all is well - thanks to all here for lending a hand.

banool commented 1 year ago

I was using the fork from @UnusualPi but it looks like they deleted it. I made those same changes in my own fork here: https://github.com/banool/personalcapital..

haochi / personalcapital

PC API login/auth errors starting 11/17/2021 #20

code object from

created

wrote

code object from

created

wrote

destroy cloudscraper

destroy personalcapital.personalcapital

destroy personalcapital

code object from

created

wrote

code object from

created

wrote

destroy cloudscraper

destroy personalcapital.personalcapital

destroy personalcapital