Not working now !!!

[kittinan]

Not working now

[kittinan]

Thai post tracking system check web browser javascript enable now. The javascript checker will create some cookies and javascript code that obfuscation, it hard to understand.

It is a high possibility we move to use Chrome headless instead CURL

headless-chromium-php

[boonva]

Tried with chrome headless, with requests more than about 50++ Thai post will now block your IP.

The requested URL was rejected. Please consult with your administrator. Your support ID is: 15299578768039961978

[kittinan]

@boonva :sob: :sob: :sob:

[boonva]

the limit is about 50 requests, the interval is not sure and the suspend duration is around 20 min.

[kittinan]

@boonva Are you use same tracking ID on 50 requests?

[boonva]

@kittinan yes, haven't try with different ID yet. Here is my script and explanation. https://medium.com/@gaingain/automate-%E0%B9%80%E0%B8%8A%E0%B9%87%E0%B8%84%E0%B8%9E%E0%B8%B1%E0%B8%AA%E0%B8%94%E0%B8%B8%E0%B9%84%E0%B8%9B%E0%B8%A3%E0%B8%A9%E0%B8%93%E0%B8%B5%E0%B8%A2%E0%B9%8C%E0%B9%84%E0%B8%97%E0%B8%A2%E0%B8%94%E0%B9%89%E0%B8%A7%E0%B8%A2-puppeteer-5d6a33d187be

[kittinan]

@boonva Thank you for your research and blog post, I will re-implement for PHP on my free time

[boonva]

Tried with 7 tracking ID with 10 requests each. got block too at around 68-70 requests

[kittinan]

Wow, it better from the last experiment. Did you add sleep function after each request?

[siamkreative]

Thanks @boonva for your solution :) I might try to implement it on my project https://github.com/SiamKreative/STS

It really pisses me off that ThailandPost does not offer any developer API. Maybe because they know their own site sucks...

[kittinan]

@boonva @SiamKreative This repo implemented Thailand post private API.

https://github.com/itsKaynine/thailand-post

[boonva]

@kittinan This is the script I used to use before it's broken.

[kittinan]

@boonva :sob: :sob: :sob:

[webhunter69]

Any success guys?

I’m really pissed off that the API they have doesn’t work and I tried all methods :(

Currently looking into iOS and Android files after decompiling them, bunch of useful stuff there, including private keys, but I’m scared they will block my server ip (or simply change their mobile app keys in future update). So I have little faith in this.

Have you guys tried emulating it on own server with JS client(s)? Without any API usage, just the normal track trace url, that comes with the slider.

Cheers from Hua Hin

[boonva]

@hansje69 I wrote my blog on using puppeteer. For ip blocking problem, you can use torr and keep changing ip. https://medium.com/@gaingain/automate-%E0%B9%80%E0%B8%8A%E0%B9%87%E0%B8%84%E0%B8%9E%E0%B8%B1%E0%B8%AA%E0%B8%94%E0%B8%B8%E0%B9%84%E0%B8%9B%E0%B8%A3%E0%B8%A9%E0%B8%93%E0%B8%B5%E0%B8%A2%E0%B9%8C%E0%B9%84%E0%B8%97%E0%B8%A2%E0%B8%94%E0%B9%89%E0%B8%A7%E0%B8%A2-puppeteer-5d6a33d187be

[webhunter69]

Wow. Very interesting Khun @boomva Too bad I can’t read Thai though (only speak). I’m luk kreung lol Nice solution by the way for IP changes!

[kittinan]

@boonva Is it possible to open-source your NodeJS module?

[boonva]

All the code (puppeteer + express nodejs) is already included in the last part of the blog kub. Feel free to use and modify. I don't have plan to improve it with Tor ip changing yet.

[kittinan]

@boonva sorry for the wrong question, I mean you should create NodeJS module for everyone use. :smiley:

I will try to implement in PHP version using headless-chromium-php

[webhunter69]

i got it working now using tor testing at thailandp???.net it was very slow, around 30secs per request now its 8-12 secs per new request which is enough for me to deploy as public (or commercial? depends if aftership.com wants to pay me for hard work lol) API if TP dont deploy it themselves. (and 1-2 secs when you request a previous trackingnumber by caching it, and periodically re-track it) im still working on it tired krup zzzz ^^

[gain9999]

@hansje69 I believe aftership has their own engine working for years kub.

[webhunter69]

@gain666 They not support a “working” version for ThailandPost “anymore” since Sep/Oct 2018.

[webhunter69]

That’s why many here in the thread fall back to find and code alternative sources and ways of gathering the carrier information :)

[kittinan]

Everyone I fixed this issue using Chrome headless follows @gain9999 solution. That working on PHP >=7.0+

:tada: :tada: :tada:

Tested on Ubuntu 18.04 (Google Cloud):

• PHP - 7.2.15
• chromium-browser

Happy Makha Bucha! :pray:

[webhunter69]

I had it working already with headless chrome, but immediately I got banned, I think after 40-50 times requesting in 1 minute so the TOR workaround works perfect which I have put in auto rotate (Random entry and random exit nodes)+ I use a list of a few popular user agents so it’s now impossible to stop me.

It was a fun project, but ideally ThailandPost internal Devs should make the API active again + free.

Anyone has any idea why they not provide a developer api for everyone? Is it related to money? Server abuse? Or they plain stupid and dumb not knowing what they are doing? It really beats me... :?:

[boonva]

@kittinan Nice kub. Thank you for contribute to opensource. Does this update also handle Thaipost timeout blocking?

@hansje69 They use external contract to build the system long time ago. They did have an API but limited per request and also has to be your own company's shipping item to be allowed to track. I totally agree that they should make it public. I did sent a cold email to their executive but no response.

[kittinan]

@boonva I have not the idea to handle their block IP address but this update support proxy.

sample_proxy.php

[klaarnut]

Hello @kittinan You came to the right place at the right time. I try repeatedly of my code about 2 weeks. but it fail . Either PhantomJs , Facebook\WebDriver ,guzzle , even default php cURL . The hard part is POST data. very thank you. I will try your package KOBKHUNKRUB

[kittinan]

I tested on my home PC got ban around 53 requests (random tracker id from 49 unique)

Test on Google Cloud (micro) got ban on 20 requests.

[webhunter69]

I tested on my home PC got ban around 53 requests (random tracker id from 49 unique)

Test on Google Cloud (micro) got ban on 20 requests.

Why not use random TOR entry+exit node ja ? Instead of home pc with only random tracker id's

let me show you how it goes with TOR... its a BIT slow, because TOR config + VPS abroad for legal reasons, but I'm still working on it, to offer a ThailandPost API :)

see attachment

Url if you want to try yourself, see in footer website :)

Anyway, I really recommend to use the TOR solution I described.

20190219225710.zip

[bankjirapan]

@kittinan You try get proxy list free from many websites when Thailand Post official website block IP about 50 request and use that proxy for request or changes proxy every 10-20 request but use Thai proxy better because i try proxy from much country may be open web long time or not working.

[kittinan]

@bankjirapan Thanks for your advice, I already used Thai proxy on Travis CI test.

This is my simple API parse proxy from proxynova.com run on Google Cloud Function.

https://asia-northeast1-mr-kittinan.cloudfunctions.net/thaiproxy

from bs4 import BeautifulSoup

import requests
import re

def index(request):

    ips = load_proxy()
    return "\n".join(ips)

def load_proxy():

    url = 'https://www.proxynova.com/proxy-server-list/country-th/'
    r = requests.get(url)
    soup = BeautifulSoup(r.content)
    rows = soup.select("#bla tr")
    ips = []
    for row in rows[1:]:
        tds = row.select('td')
        if len(tds) == 0:
            continue

        sc = tds[0].find('script').text.replace("document.write(", "").replace("');", "")
        if len(sc) == 0:
            continue

        head, tail = sc.split(' + ')
        tail = tail.replace("'", "")

        head = head.replace("'.substr(8)", "'[8:]")
        head = eval(head)

        ip = f'{head}{tail}'
        port = tds[1].text.strip()

        ip_port = f"{ip}:{port}"

        ip_port = ip_port

        ips.append(ip_port)

    return ips

[bankjirapan]

This is my code for get proxy haha. Thank for discussion and sharing.

 request('https://api.getproxylist.com/proxy?country=TH', function (error, response, body) {
                 let ProxyJson = JSON.parse(body);
                 SaveProxy =  ProxyJson.ip + ":" + ProxyJson.port;
 });

[kittinan]

@bankjirapan thanks for proxy API :+1: :+1: :+1:

[kittinan]

@boonva @hansje69 @bankjirapan

Thailand POST open the API for developer https://track.thailandpost.co.th/developerGuide

I will re-implement this repo as soon as possible when I'm available

[bankjirapan]

Haha When I woke up in the morning the server was down , ThailandPOST change website and open the api which good news and bad news.