search

wkaisertexas / tiktok-uploader

Automatically ⬆️ upload TikTok videos
https://pypi.org/project/tiktok-uploader/
322 stars 79 forks source link

_clear(desc) Not Clearing Whole Caption Text Box If Title To To Long #131

Closed Jamadoo closed 3 months ago

Jamadoo commented 3 months ago

This is a issue relating to @callum-fortune 's new Pull-Request

On Lines

    # desc populates with filename before clearing
    WebDriverWait(driver, config['explicit_wait']).until(lambda driver: desc.text != '')

    _clear(desc)

    WebDriverWait(driver, config['explicit_wait']).until(lambda driver: desc.text == '')

If the title of the file is longer then the half of the text box, only the text before the half of the text box is removed. Considor sending "Keys.End" before calling clear

    # desc populates with filename before clearing
    WebDriverWait(driver, config['explicit_wait']).until(lambda driver: desc.text != '')

    desc.send_keys(Keys.END)
    _clear(desc)

    WebDriverWait(driver, config['explicit_wait']).until(lambda driver: desc.text == '')

Works great in my experience.

I also noticed a log is send in the console of the browser ({isSameSearchValue: false}) each time the dialog pops up showcasing the available hashtags and mentions. We could use a wait until this log gets written instead of

if word[0] == "#":
                desc.send_keys(word)
                desc.send_keys(' ' + Keys.BACKSPACE)
                time.sleep(config['implicit_wait']) # Here we would add the wait until the log gets written, aka the menu gets opened. Which will speed up the hashtag adding progress a ton, because the menu pops up between 0.5-2 seconds in my experience. Thus, here we wont wait any  unnecessary time 
                desc.send_keys(Keys.ENTER)
callum-fortune commented 3 months ago

@Jamadoo You might be onto something here, I'm just applying the changes you suggested and testing them then I'll add them to my PR. For some reason though, I cannot seem to replicate the error about clearing the description, even when I provide a really long filename, it gets entirely removed for me. Could you show me what filename you are passing in?...

Jamadoo commented 3 months ago

@callum-fortune It was something along the lines "you forgot your cocaine at the Whitehouse, what's your next move_" (it only sounds weird with no context). When I ran the script the textbox gets clicked, but in the middle of the textbox so the cursor is in the middle of the textbox. Then clear gets called which only presses backspace. Thus leaving the text after the cursor in tacit and putting the script in a infinite loop waiting for the textbox to be empty.

The script is a bit unpredictable some times in my experience, so it could be one of those instances. Might as well add the keys.end as a safety procedure

callum-fortune commented 3 months ago

@Jamadoo Thanks for that, I have pushed a new commit to my outstanding PR (https://github.com/wkaisertexas/tiktok-uploader/pull/128) which contains the Keys.END you suggested. Additionally I finally found an element to anchor to when adding hashtags and mentions which has greatly increased the speed of the description insertion. I tried to use the console log that you recommended but it's really quite difficult to identify them. There is still a time.sleep(2) because I noticed a couple of anomalies when there was no wait time but I can definitely see a great difference in the speed after these changes.

I had only just woken up when I read your issue and got started so I could use some cocaine right now. Sadly though, I seem to have left it somewhere...

Jamadoo commented 3 months ago

@callum-fortune I got a suggestion from gpt how to monitor the logs and wait for a print. I'll see what I can implement tmr. He created a new class and used the WaitWebDriver to wait for a console write

callum-fortune commented 3 months ago

@Jamadoo I must confess, I tried one or two ideas from chatGPT but none of them pulled through. I also think there could be cross-browser discrepancies between how the logs are read which could cause issues in the future. With that said though, go for it, I'd be interested to see how it works in the end.

Jamadoo commented 3 months ago

@callum-fortune yea, the suggestion is focused on the chrome driver. So if anything, we can enable the log wait on chorme or else the method you mentioned. Then we just say "we recommend using Chrome driver". I can't really see why you would want to use another browser tbh

Jamadoo commented 3 months ago

I found a potential way to check the logs in chromium based browsers. We could also just remove support to Firefox 😂😭

callum-fortune commented 3 months ago

@Jamadoo I think the recommendation is already to use Chrome based browsers so yeh it would be dumb to use anything else, I guess I'm trying to cover all bases really. I'm looking forward to seeing a working example 😃. I've dropped you a follow for the help.

Jamadoo commented 3 months ago

@callum-fortune Im looking into this wait for a log thing. I got it working to get all browser logs and wait for out specific log. Then I realized that this log is send through JavaScript. And according to gpt and some of my own research you need to set up fucking websockets and shit to capture this. I aint doing allat. Ill be looking at other ways to detect when the menu pops up.

callum-fortune commented 3 months ago

@Jamadoo Yeh thats the thing, it's so extra. But the PR that is currently open (https://github.com/wkaisertexas/tiktok-uploader/pull/128) I found a way to detect the menu popup and it works fine so currently there is no need to look at the logs anyway. Maybe in the future as the UI changes more that would be necessary... who knows. Hopefully @wkaisertexas can take a look at the new PR and we can get it merged. Initially I was planning to use this project for a personal side-project but I'm quite enjoying looking into and fixing various issues tbh.

For reference, the change I made to detect the popup is in config.toml and now references the popup using the following selector:

mention_box = "//div[contains(@class, 'mention-list-popover')]"

Despite the name, mention-list-popover is used for both hashtags and account mentions

Jamadoo commented 3 months ago

@callum-fortune Oh, thats quite literally what ive been looking for.... I went into the site's javascript and found they are setting the css data of a element in the document, but could not for the life of me find it. I believe they toggle it's visibility (display type).

In the PR, i see you create the EC.presence_of_element_located, but never wait for it (Note, i code in c# not python), is this correct or should there be a WebDriverWait? image

Jamadoo commented 3 months ago

Also, @callum-fortune I found on slower internet connecitons ['selectors']['upload']['upload_video'] is not immediately accessible inside the site and cant be send keys to on _set_video. I found just adding another wait before sending the keys fixed this

            _change_to_upload_iframe(driver)

            # Wait For Input File
            driverWait = WebDriverWait(driver, config['explicit_wait'])
            upload_boxWait = EC.presence_of_element_located(
                (By.XPATH, config['selectors']['upload']['upload_video'])
                )
            driverWait.until(upload_boxWait)
Jamadoo commented 3 months ago

@callum-fortune Just tested the hashtag adding code with 

if word[0] == "#":
                logger.debug(green('- Adding Hashtag: ' + word))
                desc.send_keys(word)
                desc.send_keys(' ' + Keys.BACKSPACE)
                driverWait.until(EC.presence_of_element_located(
                    (By.XPATH, config['selectors']['upload']['mention_box'])
                ))
                time.sleep(0.5) # For Safety
                desc.send_keys(Keys.ENTER)

Works better then i ever expected

callum-fortune commented 3 months ago

@Jamadoo As per your first comment, You're totally right and I completely overlooked putting the element selector in a webDriverWait, I'm just sorting that now. As for the explicit wait for slow internet connections I'll add that to the PR too

Jamadoo commented 3 months ago

@callum-fortune The time.sleep(0.5) # For Safety is not even needed. I tested it with my shitty ass 500 kb/s mobile wifi, and it still timed the keys.enter to the millisecond. Thats all we needed

callum-fortune commented 3 months ago

@Jamadoo That's great I just tried it too and it's super fast. There is an issue with the @ mentions though where if I remove the various time.sleep(1) it doesn't load the correct account in time and selects an incorrect one. Either we keep the time.sleeps in there or figure something out...

It only seems to happen if I run the script after it hasn't ran for a while, like the cache has been cleared meaning it takes longer to load the accounts in the dropdown

Jamadoo commented 3 months ago

@callum-fortune Is the mention_box element a flex object? If it is, we can look at the amount of elements/children of the flex object and just wait untill theres more then two. I cant for the life of me find where the hell this mention_box element is, so could you also show me your inspect element window of where this mf is 😭

callum-fortune commented 3 months ago

@Jamadoo I forgot to mention finding the mention box was so fucking hard. The issue is that when it is displayed, you can't click on the dev tools because the mention box will disappear. You can paste this into the console:

setTimeout(function() {
  debugger;
}, 3000);

But time it right so that it runs the debugger whilst the mention_box is open. Then you will be able to inspect it all you want.

Jamadoo commented 3 months ago

@callum-fortune Thanks, ill quickly look into its children

Jamadoo commented 3 months ago

image @callum-fortune 🤗 looks like we got a div with children

Jamadoo commented 3 months ago

This div only appear AFTER all the names are loaded. So we just need to wait untill this div appear inside the Popover element

callum-fortune commented 3 months ago

@Jamadoo Which div would that be? When I watch the script running, I see that several users come up but not one that is an exact match of the mention in the description, it takes a few secs for the exact match to load in, meaning it ends up selecting whichever account is at the top of the list at that moment

Jamadoo commented 3 months ago

@callum-fortune On my side, when i type in any known taken username, tiktok gives me a loading animation. In this loading animation the div (with no attributes) is not displayed yet. Only when the usernames are loaded the div gets shown, and the exact match is at the top. Ill make a quick video with a username i have not searched yet

Jamadoo commented 3 months ago

@callum-fortune Look here. My first attampt, you can see i search a random tiktok username and the exact match comes up right after the loading completes. Then the console gets paused, and we can see theres a div with all the users in. Then my 3rd attempt (ignore me struggling the the second attempt) while the usernames are loading the div is not created yet. https://gofile.io/d/kfez70

callum-fortune commented 3 months ago

@Jamadoo I think we can just create a wait for the element in the popover that contains the username we are searching for since ti is displayed as a heading or whatever. I think this was done in the original implementation for this repo but when I refactored one of the big functions this would have been lost

Jamadoo commented 3 months ago

image We could, here is where the username is stored. I do see this as unnecessary tho? Or is it cause my cache is already loaded for the usernames to match first time?

callum-fortune commented 3 months ago

@Jamadoo Yeh I think it is a caching issue because 9/10 times it works perfectly for me as well but it is probably worth making the change. Also, the code would need to reference the user-id field not the name which creates another issue... check the contents of the user-id span...

Jamadoo commented 3 months ago

image @callum-fortune this is how the userId looks, we can just search for the user-id class, then select the first child. Then repeat this untill the first one is out user. And have a timeout of 5 seconds, we select the first one (we assume the user gave a invalid username)

Jamadoo commented 3 months ago

@callum-fortune Got code working to wait for the div and then wait for 2 children to appear

# Wait For Popup
                Popup = driverWait.until(EC.presence_of_element_located(
                    (By.XPATH, config['selectors']['upload']['mention_box'])
                ))
                # Wait For Div With Users
                Div = driverWait.until(
                    lambda driver: Popup.find_element(By.XPATH, "./div[not(@*)]")
                )
                # Wait until it has at least two children
                driverWait.until(
                    lambda driver: len(Div.find_elements(By.XPATH, "./*")) >= 2
                )
                logger.debug(green('- Loaded Users'))
callum-fortune commented 3 months ago

@Jamadoo What I interpreted from your message above reading 'we can just search for the user-id class, then select the first child' seems like the better idea, I'm just testing that and then we can compare

Jamadoo commented 3 months ago

@callum-fortune got my code mostly working for my idea. Just some bugs to get killed

Jamadoo commented 3 months ago

@callum-fortune Chatgpt really saving my ass rn ngl

callum-fortune commented 3 months ago

@Jamadoo With your implementation, what will happen if the search is for say "bbc" and someone has the username "abbca". Will your code only match it exactly and return the correct element for @bbc or will it return others that contain the string and perhaps return @abbca

Jamadoo commented 3 months ago

@callum-fortune In my code im looking for a exact match (not case sens). If non are found in 5 secs after the usernames are loaded. I select the top username

Jamadoo commented 3 months ago

I got some working code, testing everything rn

callum-fortune commented 3 months ago

@Jamadoo Yeh that sounds much better than my implementation tbh

Jamadoo commented 3 months ago

@callum-fortune I got some working code. You can remove the prints. It was just used for debugging

 elif word[0] == "@":
                logger.debug(green('- Adding Mention: ' + word))
                desc.send_keys(word)
                desc.send_keys(' ' + Keys.BACKSPACE)
                # Wait For Popup
                Popup = driverWait.until(EC.presence_of_element_located(
                    (By.XPATH, config['selectors']['upload']['mention_box'])
                ))
                # Wait For Div With Users
                Div = driverWait.until(
                    lambda driver: Popup.find_element(By.XPATH, "./div[not(@*)]")
                )
                # Wait until it has at least two children
                driverWait.until(
                    lambda driver: len(Div.find_elements(By.XPATH, "./*")) >= 2
                )
                ## Wait For Correct User To Appear
                # Variable indicating whether the loop should continue
                found = False
                waiting_interval = 0.5
                timeout = 5
                start_time = time.time()
                # Loop until the desired condition is met
                while not found and (time.time() - start_time < timeout):
                    # Get all children with 'user-id' class
                    user_id_elements = Div.find_elements(By.CLASS_NAME, "user-id")

                    if user_id_elements:
                        # If elements are found, check the first one
                        first_user_id = user_id_elements[0]

                        # Get the inner text
                        first_user_id_text = first_user_id.text
                        username = first_user_id_text.split(" · ")[0]
                        print("Userrname: " + username + "\nSeaerch: " + word[1:])

                        # Compare with the target word
                        if username.lower == word[1:].lower:
                            found = True
                            print("Matching User found")
                        else:
                            # If not matching, wait and repeat
                            print(f"No match. Waiting for {waiting_interval} seconds...")
                            time.sleep(waiting_interval)  # Pause for the specified interval
                    else:
                        # If no 'user-id' elements are found, wait and try again
                        print(f"No elements found. Waiting for {waiting_interval} seconds...")
                        time.sleep(waiting_interval)  # Pause before checking again
                logger.debug(green('- Selecting First User'))
                desc.send_keys(Keys.ENTER)
callum-fortune commented 3 months ago

@Jamadoo Cool I gave it a try, it did work however I noticed something. I passed @bbc into the description and it kept saying no match even though it had come up. In the end it selected @bbc but only because it was the first option and it timed out...

Jamadoo commented 3 months ago

@callum-fortune jip, it was because of my line if username.lower() == word[1:].lower():

I forgot the () after lower. Im testing again rn.

Jamadoo commented 3 months ago

works perfectly now

callum-fortune commented 3 months ago

@Jamadoo Sounds good I'm running it now too

callum-fortune commented 3 months ago

@Jamadoo I have bad news, the problem seems to have persisted. I passed in @bbc and it uploaded with something totally wrong

You can see the username that it ended up uploading:

image

Jamadoo commented 3 months ago

strange, i did the same with bbc and the correct use was added

callum-fortune commented 3 months ago

This is the thing, it seems dependent on many factors, environment, internet speed, cache etc

Jamadoo commented 3 months ago

all those waits and checks i do believe we have covered most of that? Im checking the bbc example again with some other tests and ill update if i can replicate the issue

Jamadoo commented 3 months ago

image @callum-fortune bbc was selected for me....

Jamadoo commented 3 months ago

are you sure your if statements looks like:

if username.lower() == word[1:].lower():
Jamadoo commented 3 months ago

image Was uploaded with Mention as well

Jamadoo commented 3 months ago

ive tested this code with my 50 mpb/s fiber and my 500 kb/s mobile data. Seems to work perfectly on both....

callum-fortune commented 3 months ago

@Jamadoo I missed two parentheses, I added the one that you said but not one a few characters to the left...

Testing now