_clear(desc) Not Clearing Whole Caption Text Box If Title To To Long

[Jamadoo]

This is a issue relating to @callum-fortune 's new Pull-Request

On Lines

    # desc populates with filename before clearing
    WebDriverWait(driver, config['explicit_wait']).until(lambda driver: desc.text != '')

    _clear(desc)

    WebDriverWait(driver, config['explicit_wait']).until(lambda driver: desc.text == '')

If the title of the file is longer then the half of the text box, only the text before the half of the text box is removed. Considor sending "Keys.End" before calling clear

    # desc populates with filename before clearing
    WebDriverWait(driver, config['explicit_wait']).until(lambda driver: desc.text != '')

    desc.send_keys(Keys.END)
    _clear(desc)

    WebDriverWait(driver, config['explicit_wait']).until(lambda driver: desc.text == '')

Works great in my experience.

I also noticed a log is send in the console of the browser ({isSameSearchValue: false}) each time the dialog pops up showcasing the available hashtags and mentions. We could use a wait until this log gets written instead of

if word[0] == "#":
                desc.send_keys(word)
                desc.send_keys(' ' + Keys.BACKSPACE)
                time.sleep(config['implicit_wait']) # Here we would add the wait until the log gets written, aka the menu gets opened. Which will speed up the hashtag adding progress a ton, because the menu pops up between 0.5-2 seconds in my experience. Thus, here we wont wait any  unnecessary time 
                desc.send_keys(Keys.ENTER)

[callumjamesfortune]

@Jamadoo You might be onto something here, I'm just applying the changes you suggested and testing them then I'll add them to my PR. For some reason though, I cannot seem to replicate the error about clearing the description, even when I provide a really long filename, it gets entirely removed for me. Could you show me what filename you are passing in?...

[Jamadoo]

@callum-fortune It was something along the lines "you forgot your cocaine at the Whitehouse, what's your next move_" (it only sounds weird with no context). When I ran the script the textbox gets clicked, but in the middle of the textbox so the cursor is in the middle of the textbox. Then clear gets called which only presses backspace. Thus leaving the text after the cursor in tacit and putting the script in a infinite loop waiting for the textbox to be empty.

The script is a bit unpredictable some times in my experience, so it could be one of those instances. Might as well add the keys.end as a safety procedure

[callumjamesfortune]

@Jamadoo Thanks for that, I have pushed a new commit to my outstanding PR (https://github.com/wkaisertexas/tiktok-uploader/pull/128) which contains the Keys.END you suggested. Additionally I finally found an element to anchor to when adding hashtags and mentions which has greatly increased the speed of the description insertion. I tried to use the console log that you recommended but it's really quite difficult to identify them. There is still a time.sleep(2) because I noticed a couple of anomalies when there was no wait time but I can definitely see a great difference in the speed after these changes.

I had only just woken up when I read your issue and got started so I could use some cocaine right now. Sadly though, I seem to have left it somewhere...

[Jamadoo]

@callum-fortune I got a suggestion from gpt how to monitor the logs and wait for a print. I'll see what I can implement tmr. He created a new class and used the WaitWebDriver to wait for a console write

[callumjamesfortune]

@Jamadoo I must confess, I tried one or two ideas from chatGPT but none of them pulled through. I also think there could be cross-browser discrepancies between how the logs are read which could cause issues in the future. With that said though, go for it, I'd be interested to see how it works in the end.

[Jamadoo]

@callum-fortune yea, the suggestion is focused on the chrome driver. So if anything, we can enable the log wait on chorme or else the method you mentioned. Then we just say "we recommend using Chrome driver". I can't really see why you would want to use another browser tbh

[Jamadoo]

I found a potential way to check the logs in chromium based browsers. We could also just remove support to Firefox 😂😭

[callumjamesfortune]

@Jamadoo I think the recommendation is already to use Chrome based browsers so yeh it would be dumb to use anything else, I guess I'm trying to cover all bases really. I'm looking forward to seeing a working example 😃. I've dropped you a follow for the help.

[Jamadoo]

@callum-fortune Im looking into this wait for a log thing. I got it working to get all browser logs and wait for out specific log. Then I realized that this log is send through JavaScript. And according to gpt and some of my own research you need to set up fucking websockets and shit to capture this. I aint doing allat. Ill be looking at other ways to detect when the menu pops up.

[callumjamesfortune]

@Jamadoo Yeh thats the thing, it's so extra. But the PR that is currently open (https://github.com/wkaisertexas/tiktok-uploader/pull/128) I found a way to detect the menu popup and it works fine so currently there is no need to look at the logs anyway. Maybe in the future as the UI changes more that would be necessary... who knows. Hopefully @wkaisertexas can take a look at the new PR and we can get it merged. Initially I was planning to use this project for a personal side-project but I'm quite enjoying looking into and fixing various issues tbh.

For reference, the change I made to detect the popup is in config.toml and now references the popup using the following selector:

mention_box = "//div[contains(@class, 'mention-list-popover')]"

Despite the name, mention-list-popover is used for both hashtags and account mentions

[Jamadoo]

@callum-fortune Oh, thats quite literally what ive been looking for.... I went into the site's javascript and found they are setting the css data of a element in the document, but could not for the life of me find it. I believe they toggle it's visibility (display type).

In the PR, i see you create the EC.presence_of_element_located, but never wait for it (Note, i code in c# not python), is this correct or should there be a WebDriverWait?

[Jamadoo]

Also, @callum-fortune I found on slower internet connecitons ['selectors']['upload']['upload_video'] is not immediately accessible inside the site and cant be send keys to on _set_video. I found just adding another wait before sending the keys fixed this

            _change_to_upload_iframe(driver)

            # Wait For Input File
            driverWait = WebDriverWait(driver, config['explicit_wait'])
            upload_boxWait = EC.presence_of_element_located(
                (By.XPATH, config['selectors']['upload']['upload_video'])
                )
            driverWait.until(upload_boxWait)

[Jamadoo]

@callum-fortune Just tested the hashtag adding code with

if word[0] == "#":
                logger.debug(green('- Adding Hashtag: ' + word))
                desc.send_keys(word)
                desc.send_keys(' ' + Keys.BACKSPACE)
                driverWait.until(EC.presence_of_element_located(
                    (By.XPATH, config['selectors']['upload']['mention_box'])
                ))
                time.sleep(0.5) # For Safety
                desc.send_keys(Keys.ENTER)

Works better then i ever expected

[callumjamesfortune]

@Jamadoo As per your first comment, You're totally right and I completely overlooked putting the element selector in a webDriverWait, I'm just sorting that now. As for the explicit wait for slow internet connections I'll add that to the PR too

[Jamadoo]

@callum-fortune The time.sleep(0.5) # For Safety is not even needed. I tested it with my shitty ass 500 kb/s mobile wifi, and it still timed the keys.enter to the millisecond. Thats all we needed

[callumjamesfortune]

@Jamadoo That's great I just tried it too and it's super fast. There is an issue with the @ mentions though where if I remove the various time.sleep(1) it doesn't load the correct account in time and selects an incorrect one. Either we keep the time.sleeps in there or figure something out...

It only seems to happen if I run the script after it hasn't ran for a while, like the cache has been cleared meaning it takes longer to load the accounts in the dropdown

[Jamadoo]

@callum-fortune Is the mention_box element a flex object? If it is, we can look at the amount of elements/children of the flex object and just wait untill theres more then two. I cant for the life of me find where the hell this mention_box element is, so could you also show me your inspect element window of where this mf is 😭

[callumjamesfortune]

@Jamadoo I forgot to mention finding the mention box was so fucking hard. The issue is that when it is displayed, you can't click on the dev tools because the mention box will disappear. You can paste this into the console:

setTimeout(function() {
  debugger;
}, 3000);

But time it right so that it runs the debugger whilst the mention_box is open. Then you will be able to inspect it all you want.

[Jamadoo]

@callum-fortune Thanks, ill quickly look into its children

[Jamadoo]

@callum-fortune 🤗 looks like we got a div with children

[Jamadoo]

This div only appear AFTER all the names are loaded. So we just need to wait untill this div appear inside the Popover element

[callumjamesfortune]

@Jamadoo Which div would that be? When I watch the script running, I see that several users come up but not one that is an exact match of the mention in the description, it takes a few secs for the exact match to load in, meaning it ends up selecting whichever account is at the top of the list at that moment

[Jamadoo]

@callum-fortune On my side, when i type in any known taken username, tiktok gives me a loading animation. In this loading animation the div (with no attributes) is not displayed yet. Only when the usernames are loaded the div gets shown, and the exact match is at the top. Ill make a quick video with a username i have not searched yet

[Jamadoo]

@callum-fortune Look here. My first attampt, you can see i search a random tiktok username and the exact match comes up right after the loading completes. Then the console gets paused, and we can see theres a div with all the users in. Then my 3rd attempt (ignore me struggling the the second attempt) while the usernames are loading the div is not created yet. https://gofile.io/d/kfez70

[callumjamesfortune]

@Jamadoo I think we can just create a wait for the element in the popover that contains the username we are searching for since ti is displayed as a heading or whatever. I think this was done in the original implementation for this repo but when I refactored one of the big functions this would have been lost

[Jamadoo]

We could, here is where the username is stored. I do see this as unnecessary tho? Or is it cause my cache is already loaded for the usernames to match first time?

[callumjamesfortune]

@Jamadoo Yeh I think it is a caching issue because 9/10 times it works perfectly for me as well but it is probably worth making the change. Also, the code would need to reference the user-id field not the name which creates another issue... check the contents of the user-id span...

[Jamadoo]

@callum-fortune this is how the userId looks, we can just search for the user-id class, then select the first child. Then repeat this untill the first one is out user. And have a timeout of 5 seconds, we select the first one (we assume the user gave a invalid username)

[Jamadoo]

@callum-fortune Got code working to wait for the div and then wait for 2 children to appear

# Wait For Popup
                Popup = driverWait.until(EC.presence_of_element_located(
                    (By.XPATH, config['selectors']['upload']['mention_box'])
                ))
                # Wait For Div With Users
                Div = driverWait.until(
                    lambda driver: Popup.find_element(By.XPATH, "./div[not(@*)]")
                )
                # Wait until it has at least two children
                driverWait.until(
                    lambda driver: len(Div.find_elements(By.XPATH, "./*")) >= 2
                )
                logger.debug(green('- Loaded Users'))

[callumjamesfortune]

@Jamadoo What I interpreted from your message above reading 'we can just search for the user-id class, then select the first child' seems like the better idea, I'm just testing that and then we can compare

[Jamadoo]

@callum-fortune got my code mostly working for my idea. Just some bugs to get killed

[Jamadoo]

@callum-fortune Chatgpt really saving my ass rn ngl

[callumjamesfortune]

@Jamadoo With your implementation, what will happen if the search is for say "bbc" and someone has the username "abbca". Will your code only match it exactly and return the correct element for @bbc or will it return others that contain the string and perhaps return @abbca

[Jamadoo]

@callum-fortune In my code im looking for a exact match (not case sens). If non are found in 5 secs after the usernames are loaded. I select the top username

[Jamadoo]

I got some working code, testing everything rn

[callumjamesfortune]

@Jamadoo Yeh that sounds much better than my implementation tbh

[Jamadoo]

@callum-fortune I got some working code. You can remove the prints. It was just used for debugging

 elif word[0] == "@":
                logger.debug(green('- Adding Mention: ' + word))
                desc.send_keys(word)
                desc.send_keys(' ' + Keys.BACKSPACE)
                # Wait For Popup
                Popup = driverWait.until(EC.presence_of_element_located(
                    (By.XPATH, config['selectors']['upload']['mention_box'])
                ))
                # Wait For Div With Users
                Div = driverWait.until(
                    lambda driver: Popup.find_element(By.XPATH, "./div[not(@*)]")
                )
                # Wait until it has at least two children
                driverWait.until(
                    lambda driver: len(Div.find_elements(By.XPATH, "./*")) >= 2
                )
                ## Wait For Correct User To Appear
                # Variable indicating whether the loop should continue
                found = False
                waiting_interval = 0.5
                timeout = 5
                start_time = time.time()
                # Loop until the desired condition is met
                while not found and (time.time() - start_time < timeout):
                    # Get all children with 'user-id' class
                    user_id_elements = Div.find_elements(By.CLASS_NAME, "user-id")

                    if user_id_elements:
                        # If elements are found, check the first one
                        first_user_id = user_id_elements[0]

                        # Get the inner text
                        first_user_id_text = first_user_id.text
                        username = first_user_id_text.split(" · ")[0]
                        print("Userrname: " + username + "\nSeaerch: " + word[1:])

                        # Compare with the target word
                        if username.lower == word[1:].lower:
                            found = True
                            print("Matching User found")
                        else:
                            # If not matching, wait and repeat
                            print(f"No match. Waiting for {waiting_interval} seconds...")
                            time.sleep(waiting_interval)  # Pause for the specified interval
                    else:
                        # If no 'user-id' elements are found, wait and try again
                        print(f"No elements found. Waiting for {waiting_interval} seconds...")
                        time.sleep(waiting_interval)  # Pause before checking again
                logger.debug(green('- Selecting First User'))
                desc.send_keys(Keys.ENTER)

[callumjamesfortune]

@Jamadoo Cool I gave it a try, it did work however I noticed something. I passed @bbc into the description and it kept saying no match even though it had come up. In the end it selected @bbc but only because it was the first option and it timed out...

[Jamadoo]

@callum-fortune jip, it was because of my line if username.lower() == word[1:].lower():

I forgot the () after lower. Im testing again rn.

[Jamadoo]

works perfectly now

[callumjamesfortune]

@Jamadoo Sounds good I'm running it now too

[callumjamesfortune]

@Jamadoo I have bad news, the problem seems to have persisted. I passed in @bbc and it uploaded with something totally wrong

You can see the username that it ended up uploading:

[Jamadoo]

strange, i did the same with bbc and the correct use was added

[callumjamesfortune]

This is the thing, it seems dependent on many factors, environment, internet speed, cache etc

[Jamadoo]

all those waits and checks i do believe we have covered most of that? Im checking the bbc example again with some other tests and ill update if i can replicate the issue

[Jamadoo]

@callum-fortune bbc was selected for me....

[Jamadoo]

are you sure your if statements looks like:

if username.lower() == word[1:].lower():

[Jamadoo]

Was uploaded with Mention as well

[Jamadoo]

ive tested this code with my 50 mpb/s fiber and my 500 kb/s mobile data. Seems to work perfectly on both....

[callumjamesfortune]

@Jamadoo I missed two parentheses, I added the one that you said but not one a few characters to the left...

Testing now