Open deepernewbie opened 3 months ago
Could you please clarify the issue?
Hi there,
This is due to the EU Consent Request page. (See #1).
As mentioned in the issue by IHannes, they added a cookie and everything seems to be working fine.
We can add the CONSENT
cookie and theoretically it should be working as expected.
cookies = { "CONSENT": "YES+" }
# tag: v0.3
get_flights(filter, cookies=cookies)
Refer to stackoverflow.com • bypassing-eu-consent-request for more.
P.S. I'm not aware of the "EU Consent Request" when developing this project since I'm not in the region. I'm deeply sorry for my oversight and any inconvenience caused.
Regards, AWeirdDev
No this is not about the cookies I know about this because I debugged into the return request and get the html text and rendered it in
https://htmledit.squarefree.com/
it renders find but there are no flights and blue color flow indicating search is continuing is running left to right I believe this is about requests.get does not wait for an object to load and returns immediately If I use the url generated by your code on some other google flight scraper that uses selenium It works because they added something like as follows
driver.get(url)
WebDriverWait(driver, timeout=10).until(lambda d: len(_get_flight_elements(d)) > 100)
results = _get_flight_elements(driver)
where
def _get_flight_elements(driver):
return driver.find_element(by=By.XPATH, value='//body[@id = "yDmH0d"]').text.split('\n')
Hi again,
Ahh, I understand the context now.
Both round-trip
and one-way
works fine, but not multi-city
, as cited here:
I assume that multi-city
loads custom Javascript that cannot be handled by requests
. It requires something like puppeteer
or playwright
.
Just to keep this package light, we can use the try.playwright.tech API to scrape the flights. If you'd like me to implement this, please let me know. Alternatively, you can try the code below before v0.4
rolls out:
```python requests.post( "https://try.playwright.tech/service/control/run", headers={ "User-Agent": "... your user agent" }, json={ "code": "# some playwright python code", "language": "python" } ) ```
Sincerely, AWeirdDev
@AWeirdDev dude this project is awesome but what's even more awesome is your readme. Bro you could like write a newsletter love your style, keep up the great work! I'm working on trying to extract prices, did you look into this / make any progress or hit any walls I should know of? No luck yet, found some prices in the html but they are super random and all over a thousand dollars haha.
Hey @JobeOneKenobi,
Prices are now available on v1.x
. For more information, please refer to wiki/What's-New.
P.S. I do put a lot of time into the READMEs, and you're the first to perceive this :skull: Thanks, man!
Cheers
Awesome!
Wait bro are your initials JC? Cuz SAME lol
On Sat, Jul 13, 2024 at 11:31 PM JC @.***> wrote:
Hey @JobeOneKenobi https://github.com/JobeOneKenobi,
Prices are now available on v1.x. For more information, please refer to wiki/What's-New https://github.com/AWeirdDev/flights/wiki/What's-New.
P.S. I do put a lot of time into the READMEs, and you're the first to perceive this 💀 Thanks, man!
Cheers
— Reply to this email directly, view it on GitHub https://github.com/AWeirdDev/flights/issues/2#issuecomment-2227182609, or unsubscribe https://github.com/notifications/unsubscribe-auth/BDGRANGHKDEY6KJRI3HJV23ZMHWKVAVCNFSM6AAAAABIG7AQR2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRXGE4DENRQHE . You are receiving this because you were mentioned.Message ID: @.***>
returns no flights however if I manually use
https://www.google.com/travel/flights?tfs=GhoSCjIwMjQtMDYtMTlqBRIDSVNUcgUSA0xBWBoaEgoyMDI0LTA3LTA5agUSA05ZQ3IFEgNJU1RCAQFIAZgBAw%3D%3D&hl=en&tfu=EgQIABABIgA
which is the generated link there are flights
I think the problem is results take about 2-3 secs to populate but response.get returns almost immediately without waiting