ArthurG / Kijiji-Repost-Headless

Repost / Post Kijiji ads with Python
MIT License
144 stars 59 forks source link

"LOGIN_RECAPTCHA_FAIL" came back #198

Open AlexZhouAA opened 3 years ago

AlexZhouAA commented 3 years ago

It worked flawlessly until Oct. 6, but suddenly collapsed with the following error {"errors":[{"message":"Something went wrong during authentication.","locations":[{"line":2,"column":3}],"path":["loginUser"],"extensions":{"errorCode":"LOGIN_RECAPTCHA_FAIL","statusCode":401}}],"data":{"loginUser":null}}

I'm using the latest master, please help

nconnector commented 3 years ago

Same issue

PescoTan commented 3 years ago

Same issue, occurred oct 6th

encoad commented 3 years ago

Since sessions seem to stay alive forever, maybe we can take a valid logged in session, steal the cookies and dump them into Kijiji-Repost-Headless?

iparamr commented 3 years ago

Same issue

Aluui commented 3 years ago

@encoad I like your idea. @ArthurG just point us in the right direction. I would be happy to fix this and create a PR

Rings4 commented 3 years ago

Same here. This tool has been incredibly unreliable since summer. Even when it did work, it would be very hit or miss. Doesn't help that sometimes kijiji itself takes a couple days to actually purge a deleted ad from the system before you're allowed to post it again.. Really hoping this can be fixed, used to work great last year!

IH3lios commented 3 years ago

Any hope of getting this going again?

jackm commented 3 years ago

I think the fundamental issue here is the way the login is done - it is trying to emulate a user on a web browser but once captcha is involved it makes this method much more difficult. Using something like the backend Kijiji API for login would be more robust and a less fail-prone method.

JohnBakery commented 3 years ago

Since sessions seem to stay alive forever, maybe we can take a valid logged in session, steal the cookies and dump them into Kijiji-Repost-Headless?

That is a good idea. I just hardcoded session id cookie into the code and it's posting ads with no issue. If anyone wants to do the same thing, here is what I did.

kijiji_api.py

  1. Add a new line after line #98 and type """
  2. Add a new line after line #118 and type """
  3. After line #119 add the following code

cookie_dict = {'ssid': 'REPLACE_WITH_YOUR_SESSION_ID'} requests.utils.add_dict_to_cookiejar(self.session.cookies,cookie_dict)

To get your session id.

  1. Go to kijiji.ca
  2. Login into your account if you are not logged in.
  3. Press F12
  4. In FireFox: Go to Storage. In Chrome: Go to Application.
  5. Open up Cookies, select https://www.kijiji.ca, look for a cookie "ssid", double click on the value field and copy paste it into the code where it says REPLACE_WITH_YOUR_SESSION_ID.

Save kijiji_api.py, run it.

AlexZhouAA commented 3 years ago

@JohnBakery, thank you for your elaborated procedures first. I followed your steps and got an error: NameError: name 'resp' is not defined Then I realized there're 8 resp = ... statements within the file kijiji_api.py. Sorry, I'm not a python developer, do I need do the same to the rest?

meeoh commented 3 years ago

Since sessions seem to stay alive forever, maybe we can take a valid logged in session, steal the cookies and dump them into Kijiji-Repost-Headless?

That is a good idea. I just hardcoded session id cookie into the code and it's posting ads with no issue. If anyone wants to do the same thing, here is what I did.

kijiji_api.py

  1. Comment out line #100 and #117. (Add # in front of those line)
  2. After line #117 add the following code

cookie_dict = {'ssid': 'REPLACE_WITH_YOUR_SESSION_ID'} requests.utils.add_dict_to_cookiejar(self.session.cookies,cookie_dict)

To get your session id.

  1. Go to kijiji.ca
  2. Login into your account if you are not logged in.
  3. Press F12
  4. In FireFox: Go to Storage. In Chrome: Go to Application.
  5. Open up Cookies, select https://www.kijiji.ca, look for a cookie "ssid", double click on the value field and copy paste it into the code where it says REPLACE_WITH_YOUR_SESSION_ID.

Save kijiji_api.py, run it.

If we just do these things via selenium, then shut it down that should work fine. And since the session stays alive forever its just a one-time thing, so there could be a check for existance before having to trigger selenium stuff.

Either that or we add @JohnBakery's instructions to the readme which also isnt that bad tbh

JohnBakery commented 3 years ago

@AlexZhouAA , my bad. In my code I actually commented out the whole block of code, but for the reply I only mentioned two lines to make the instructions easier, completely missing that resp is used on another line in the login function.

I edited my original post with the correct instructions. Do not comment out resp anywhere else.

AlexZhouAA commented 3 years ago

@JohnBakery, after I commented out the whole block as you instructed, I still got the error: File "~\Kijiji-Repost-Headless-master\kijiji_repost_headless\kijiji_api.py", line 124, in login raise KijijiApiException("Could not log in.", resp.text) NameError: name 'resp' is not defined

And then I commented out the following

if not self.is_logged_in()
     raise KijijiApiException("Could not log in.", resp.text)

and try gain, I got error as File "~\Kijiji-Repost-Headless-master\kijiji_repost_headless\kijiji_api.py", line 251, in get_all_ads ads_json = json.loads(resp.text)

Please help & thanks for your time

JohnBakery commented 3 years ago

@AlexZhouAA this means your session id value is incorrect. Either you didn't copy it completely (mine is over 300 characters long) or it expired. Make sure you when you login, select "Remember Me" option

AlexZhouAA commented 3 years ago

@JohnBakery, you are absolutely right. I did copy the wrong ssid. Thank you so much. This time I got all text part posted successfully, but all images failed to be uploaded

Image #1 upload failed on try #1
Image #1 upload failed on try #2
Image #1 upload failed on try #3
Image #2 upload failed on try #1
Image #2 upload failed on try #2
Image #2 upload failed on try #3
Image #3 upload failed on try #1
Image #3 upload failed on try #2
Image #3 upload failed on try #3
Image #4 upload failed on try #1
Image #4 upload failed on try #2
Image #4 upload failed on try #3
Traceback (most recent call last):
  File "~\Python36\lib\runpy.py", line 193, 
        in _run_module_as_main "__main__", mod_spec)
  File "~\Python36\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "~\Kijiji-Repost-Headless-master\kijiji_repost_headless\__main__.py", line 209, 
        in <module> main()
  File "~\Kijiji-Repost-Headless-master\kijiji_repost_headless\__main__.py", line 51, in main
    args.function(args)
  File "~\Kijiji-Repost-Headless-master\kijiji_repost_headless\__main__.py", line 98, in post_ad
    api.post_ad_using_data(data, image_files)
  File "~\Kijiji-Repost-Headless-master\kijiji_repost_headless\kijiji_api.py", line 238, in post_ad_using_data
    raise KijijiApiException("Could not post ad.", resp.text)
kijiji_api.KijijiApiException: Could not post ad.
JohnBakery commented 3 years ago

@AlexZhouAA Not sure why your images are failing. Maybe they are too big in size?

AlexZhouAA commented 3 years ago

@JohnBakery, I have used the same scripts with the same images to post ads on kijiji.ca for a half year. It's not because of the image sizes for sure. Thank you so much for the helps

AminDhouib commented 3 years ago

After reading @JohnBakery 's solution, I tried it and I managed to post an advertisement with pictures! It works!

Oddly enough when I used this app before every image would be uploaded on its first try but now it takes usually 2 tries for it to be uploaded. Addtionally, even when the ad has been uploaded successfully I still get this error::

Traceback (most recent call last):
  File "~\Python36\lib\runpy.py", line 193, 
        in _run_module_as_main "__main__", mod_spec)
  File "~\Python36\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "~\Kijiji-Repost-Headless-master\kijiji_repost_headless\__main__.py", line 209, 
        in <module> main()
  File "~\Kijiji-Repost-Headless-master\kijiji_repost_headless\__main__.py", line 51, in main
    args.function(args)
  File "~\Kijiji-Repost-Headless-master\kijiji_repost_headless\__main__.py", line 98, in post_ad
    api.post_ad_using_data(data, image_files)
  File "~\Kijiji-Repost-Headless-master\kijiji_repost_headless\kijiji_api.py", line 238, in post_ad_using_data
    raise KijijiApiException("Could not post ad.", resp.text)
kijiji_api.KijijiApiException: Could not post ad.

It could be a false error where it raises an Exception when it should not. But, I can confirm that @JohnBakery 's solution is working and we could build upon his or her solution. So we are getting somewhere! ^_^

prgrm commented 3 years ago

Since sessions seem to stay alive forever, maybe we can take a valid logged in session, steal the cookies and dump them into Kijiji-Repost-Headless?

That is a good idea. I just hardcoded session id cookie into the code and it's posting ads with no issue. If anyone wants to do the same thing, here is what I did.

kijiji_api.py

1. Add a new line after line #98 and type """

2. Add a new line after line #118 and type """

3. After line #119 add the following code

cookie_dict = {'ssid': 'REPLACE_WITH_YOUR_SESSION_ID'} requests.utils.add_dict_to_cookiejar(self.session.cookies,cookie_dict)

To get your session id.

1. Go to kijiji.ca

2. Login into your account if you are not logged in.

3. Press F12

4. In FireFox: Go to Storage. In Chrome: Go to Application.

5. Open up Cookies, select https://www.kijiji.ca, look for a cookie "ssid", double click on the value field and copy paste it into the code where it says REPLACE_WITH_YOUR_SESSION_ID.

Save kijiji_api.py, run it.

To be clear, do I comment out the lines starting with: login_url ... and ending with: resp = ...

I did the above but some ads post properly and some have failures with image uploads. Thanks for the help.

JohnBakery commented 3 years ago

After reading @JohnBakery 's solution, I tried it and I managed to post an advertisement with pictures! It works!

Oddly enough when I used this app before every image would be uploaded on its first try but now it takes usually 2 tries for it to be uploaded. Addtionally, even when the ad has been uploaded successfully I still get this error::

Traceback (most recent call last):
  File "~\Python36\lib\runpy.py", line 193, 
      in _run_module_as_main "__main__", mod_spec)
  File "~\Python36\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "~\Kijiji-Repost-Headless-master\kijiji_repost_headless\__main__.py", line 209, 
      in <module> main()
  File "~\Kijiji-Repost-Headless-master\kijiji_repost_headless\__main__.py", line 51, in main
    args.function(args)
  File "~\Kijiji-Repost-Headless-master\kijiji_repost_headless\__main__.py", line 98, in post_ad
    api.post_ad_using_data(data, image_files)
  File "~\Kijiji-Repost-Headless-master\kijiji_repost_headless\kijiji_api.py", line 238, in post_ad_using_data
    raise KijijiApiException("Could not post ad.", resp.text)
kijiji_api.KijijiApiException: Could not post ad.

It could be a false error where it raises an Exception when it should not. But, I can confirm that @JohnBakery 's solution is working and we could build upon his or her solution. So we are getting somewhere! ^_^

Change if "deleteWithoutSurvey" not in resp.text: to if "deleteSurveyReasons" not in resp.text:

Since sessions seem to stay alive forever, maybe we can take a valid logged in session, steal the cookies and dump them into Kijiji-Repost-Headless?

That is a good idea. I just hardcoded session id cookie into the code and it's posting ads with no issue. If anyone wants to do the same thing, here is what I did. kijiji_api.py

1. Add a new line after line #98 and type """

2. Add a new line after line #118 and type """

3. After line #119 add the following code

cookie_dict = {'ssid': 'REPLACE_WITH_YOUR_SESSION_ID'} requests.utils.add_dict_to_cookiejar(self.session.cookies,cookie_dict) To get your session id.

1. Go to kijiji.ca

2. Login into your account if you are not logged in.

3. Press F12

4. In FireFox: Go to Storage. In Chrome: Go to Application.

5. Open up Cookies, select https://www.kijiji.ca, look for a cookie "ssid", double click on the value field and copy paste it into the code where it says REPLACE_WITH_YOUR_SESSION_ID.

Save kijiji_api.py, run it.

To be clear, do I comment out the lines starting with: login_url ... and ending with: resp = ...

I did the above but some ads post properly and some have failures with image uploads. Thanks for the help.

Yes, that's correct. Not sure about the images. I reposted like 6 ads and none of them had the image issue

bobokun commented 3 years ago

@JohnBakery any chance you can make a pull request and have this merged back to master branch so this can be closed? I think we can add the instructions on the readme to provide the session_id as well as one of the parameters passed?

xsimo commented 3 years ago

@JohnBakery, I have used the same scripts with the same images to post ads on kijiji.ca for a half year. It's not because of the image sizes for sure. Thank you so much for the helps

Also got the same issue, the ad gets uploaded, but the success check that is based on the presence of "deleteSurveyReasons" in the resp.text is not functionning. So the main.py triggers another attempt eventhough the first was successfull. I ended up with the intended ad posted 5 times. I tried to add this line print(resp.text) just to check if the response was ok and it is. It does contain the "deleteSurveyReasons" Eventhough i am an experienced programmer, i am new to python !

yunners commented 3 years ago

Still getting errors on login with ads not being posted either:

Traceback (most recent call last): File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Users/jmshong/Downloads/Kijiji-Repost-Headless-master/kijiji_repost_headless/main.py", line 209, in main() File "/Users/jmshong/Downloads/Kijiji-Repost-Headless-master/kijiji_repost_headless/main.py", line 51, in main args.function(args) File "/Users/jmshong/Downloads/Kijiji-Repost-Headless-master/kijiji_repost_headless/main.py", line 87, in post_ad api.login(args.username, args.password) File "/Users/jmshong/Downloads/Kijiji-Repost-Headless-master/kijiji_repost_headless/kijiji_api.py", line 129, in login raise KijijiApiException("Could not log in.", resp.text) NameError: name 'resp' is not defined

ArthurG commented 3 years ago

I believe that this issue should be fixed now. Please comment if you experience further issue

iparamr commented 3 years ago

I am running the following command: python3 kijiji_repost_headless -u $USER -p $PASS -s $SSID show But I keep getting the following error:

usage: kijiji_repost_headless [-h] [-s SSID] {post,show,delete,nuke,check_ad,repost,build_ad} ...
kijiji_repost_headless: error: invalid choice: 'xxxxxxxxxx@gmail.com' (choose from 'post', 'show', 'delete', 'nuke', 'check_ad', 'repost', 'build_ad')

Last time when it used to work I would only use this command: python3 kijiji_repost_headless -u $USER -p $PASS show

I have updated the code to the last from your repo.

What am I doing wrong here? 😢

jackm commented 3 years ago

I am running the following command: python3 kijiji_repost_headless -u $USER -p $PASS -s $SSID show But I keep getting the following error:

usage: kijiji_repost_headless [-h] [-s SSID] {post,show,delete,nuke,check_ad,repost,build_ad} ...
kijiji_repost_headless: error: invalid choice: 'xxxxxxxxxx@gmail.com' (choose from 'post', 'show', 'delete', 'nuke', 'check_ad', 'repost', 'build_ad')

Program arguments -u and -p have been removed.

iparamr commented 3 years ago

@jackm Thanks! It's still there in the ReadMe so I didn't know. What's the correct command now? I am using: python3 kijiji_repost_headless -s $SSID show --key id

but that doesn't work either. I get the following error now:

Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "kijiji_repost_headless/__main__.py", line 198, in <module> main() File "kijiji_repost_headless/__main__.py", line 50, in main args.function(args) File "kijiji_repost_headless/__main__.py", line 101, in show_ads api.login(args.ssid) File "kijiji_repost_headless/kijiji_api.py", line 102, in login with open(ssid_path) as ssidFile: FileNotFoundError: [Errno 2] No such file or directory: '/Users/xxx/Code/Kijiji-Repost-Headless/xxxxxxxxx'

NEVER MIND: I found the instructions in the updated ReadMe:

Getting your SSID

Before posting an ad, you will need to manually get your Kijiji SSID.

  1. Log into Kijiji in your browser. Make sure "remember me" is checked.
  2. Inspect the cookies that have been set by Kijiji for your browser.
  3. Find the cookie named "ssid" and copy its value.
  4. Paste the value into the "ssid.txt" file.

Use the command like this: python3 kijiji_repost_headless -s ssid.txt show --key id

Rings4 commented 3 years ago

@jackm Thanks! It's still there in the ReadMe so I didn't know. What's the correct command now? I am using: python3 kijiji_repost_headless -s $SSID show --key id

but that doesn't work either. I get the following error now:

Traceback (most recent call last): File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "kijiji_repost_headless/__main__.py", line 198, in <module> main() File "kijiji_repost_headless/__main__.py", line 50, in main args.function(args) File "kijiji_repost_headless/__main__.py", line 101, in show_ads api.login(args.ssid) File "kijiji_repost_headless/kijiji_api.py", line 102, in login with open(ssid_path) as ssidFile: FileNotFoundError: [Errno 2] No such file or directory: '/Users/xxx/Code/Kijiji-Repost-Headless/xxxxxxxxx'

NEVER MIND: I found the instructions in the updated ReadMe:

Getting your SSID

Before posting an ad, you will need to manually get your Kijiji SSID.

1. Log into Kijiji in your browser. Make sure "remember me" is checked.

2. Inspect the cookies that have been set by Kijiji for your browser.

3. Find the cookie named "ssid" and copy its value.

4. Paste the value into the "ssid.txt" file.

Use the command like this: python3 kijiji_repost_headless -s ssid.txt show --key id

May I ask why you're using a different command not in the readme? The commands seem to be the same as before, only difference is to just get your ssid and paste it in ssid.txt. Am I missing something? I'd like to be clear on this before attempting to repost an ad.

iparamr commented 3 years ago

@Rings4 It's working now. Thanks

Corgano commented 3 years ago

I got my SSID using chrome developer tools, and I'm using python kijiji_repost_headless -s ssid.txt show repost !dir!\item.yml I know the path is 100% correct. Keeps giving me "Could not log in"

Also what does " show --key id" do in the parameters?