Can the server be selected when scrubbing source prices?

[RitzDaCat]

Hi,

Digging around on FFXIAH I found I could pull different servers using the following in Powershell.

` $number = $_ $Uri = "https://www.ffxiah.com/item/"+$number $Body = @{ sid = 28 } $properRequest = Invoke-WebRequest -Method HEAD -Uri $Uri

write-host $properRequest.BaseResponse.RequestMessage.RequestUri.AbsolutePath

    $newURL = "https://www.ffxiah.com" + $properRequest.BaseResponse.RequestMessage.RequestUri.AbsolutePath
    $response = Invoke-WebRequest -Uri $newURL -Form $Body -Method Post `

The SID determines the server which is this list.

1. Asura
2. Bahamut
3. Bismarck
4. Carbuncle
5. Cerberus
6. Fenrir
7. Lakshmi
8. Leviathan
9. Odin
10. Phoenix
11. Quetzalcoatl
12. Ragnarok
13. Shiva
14. Siren
15. Sylph
16. Valefor
18. Alexander
19. Caitsith
20. Diabolos
21. Fairy
22. Garuda
23. Gilgamesh
24. Hades
25. Ifrit
26. Kujata
27. Midgardsormr
28. Pandemonium
29. Ramuh
30. Remora
31. Seraph
32. Titan
33. Unicorn

I believe if this was updated to be able to choose what server that would be an awesome enhancement request. Please let me know if there's anything additional I can provide to help.

[AdamGagorik]

I will look into this feature. The problem I always had was that FFXIAH used javascript when I first developed the server and I only really had the ability to download static HTML. Without the use of something like selenium I was stuck and forced to use Bahamut.

[AdamGagorik]

@PaulAnthonyReitz

I implemented this feature and tested it. The trick was to have the item name in the URL. Otherwise FFXIAH did not respect the sid parameter. I think that is what you were doing with the HEAD request and absolute URL. You used them to somehow get the name of the item in addition to its integer ID. I ended up doing a similar thing on the Python side and it works.

For example (download from Cerberus):

(base) ➜  pydarkstar git:(master) pwd
/Users/adam/workspace/pydarkstar/bin
(base) ➜  pydarkstar git:(master) conda activate pydarkstar
(pydarkstar) ➜  bin git:(master) python ./scrub.py --threads 8 --server cerberus --itemids 0 1 2
[2021-06-26 21:05:56][MainProcess][MainThread][INFO ]: (0x7f94580b4240) Options: config     = config.yaml
[2021-06-26 21:05:56][MainProcess][MainThread][INFO ]: (0x7f94580b4240) Options: verbose    = True
[2021-06-26 21:05:56][MainProcess][MainThread][INFO ]: (0x7f94580b4240) Options: silent     = False
[2021-06-26 21:05:56][MainProcess][MainThread][INFO ]: (0x7f94580b4240) Options: stub       = items
[2021-06-26 21:05:56][MainProcess][MainThread][INFO ]: (0x7f94580b4240) Options: overwrite  = False
[2021-06-26 21:05:56][MainProcess][MainThread][INFO ]: (0x7f94580b4240) Options: backup     = True
[2021-06-26 21:05:56][MainProcess][MainThread][INFO ]: (0x7f94580b4240) Options: stock01    = 5
[2021-06-26 21:05:56][MainProcess][MainThread][INFO ]: (0x7f94580b4240) Options: stock12    = 5
[2021-06-26 21:05:56][MainProcess][MainThread][INFO ]: (0x7f94580b4240) Options: itemids    = ????
[2021-06-26 21:05:56][MainProcess][MainThread][INFO ]: (0x7f94580b4240) Options: threads    = 8
[2021-06-26 21:05:56][MainProcess][MainThread][INFO ]: (0x7f94580b4240) Options: server     = cerberus
[2021-06-26 21:05:56][MainProcess][MainThread][INFO ]: (0x7f94580b4240) Options: urls       = ????
[2021-06-26 21:05:56][MainProcess][MainThread][DEBUG]: (0x7f9458d451d0) FFXIAHScrubber: init
[2021-06-26 21:05:56][MainProcess][MainThread][DEBUG]: (0x7f9458d451d0) FFXIAHScrubber: forcing redownload of data
[2021-06-26 21:05:56][MainProcess][MainThread][DEBUG]: (0x7f9458d451d0) FFXIAHScrubber: using passed ids
[2021-06-26 21:05:56][MainProcess][MainThread][INFO ]: (0x7f9458d451d0) FFXIAHScrubber: getting data
[2021-06-26 21:05:56][MainProcess][MainThread][INFO ]: (0x7f9458d451d0) FFXIAHScrubber: executing in parallel with threads=8
[2021-06-26 21:05:56][MainProcess][ExThread_0][DEBUG]: (0x7f9458d451d0) FFXIAHScrubber: open server=cerberus (000000/000003,  0.00) http://www.ffxiah.com/item/0
[2021-06-26 21:05:56][MainProcess][ExThread_1][DEBUG]: (0x7f9458d451d0) FFXIAHScrubber: open server=cerberus (000001/000003, 33.33) http://www.ffxiah.com/item/1
[2021-06-26 21:05:56][MainProcess][ExThread_2][DEBUG]: (0x7f9458d451d0) FFXIAHScrubber: open server=cerberus (000002/000003, 66.67) http://www.ffxiah.com/item/2
[2021-06-26 21:05:57][MainProcess][MainThread][DEBUG]: (0x7f9458d451d0) FFXIAHScrubber: item count = 3
[2021-06-26 21:05:57][MainProcess][MainThread][DEBUG]: (0x7f9458d451d0) FFXIAHScrubber: data count = 3
[2021-06-26 21:05:57][MainProcess][MainThread][DEBUG]: (0x7f9458c9f940) ItemList: init
[2021-06-26 21:05:57][MainProcess][MainThread][DEBUG]: backup (old): /Users/adam/workspace/pydarkstar/bin/items.csv
[2021-06-26 21:05:57][MainProcess][MainThread][DEBUG]: backup (new): /Users/adam/workspace/pydarkstar/bin/items.csv.1
[2021-06-26 21:05:57][MainProcess][MainThread][INFO ]: (0x7f9458c9f940) ItemList: overwriting file...
[2021-06-26 21:05:57][MainProcess][MainThread][INFO ]: (0x7f9458c9f940) ItemList: save /Users/adam/workspace/pydarkstar/bin/items.csv
[2021-06-26 21:05:57][MainProcess][MainThread][INFO ]: exit

For example (download from Gilgamesh):

(pydarkstar) ➜  bin git:(master) ✗ python ./scrub.py --threads 8 --server gilgamesh --itemids 0 1 2
[2021-06-26 21:07:19][MainProcess][MainThread][INFO ]: (0x7f8648124240) Options: config     = config.yaml
[2021-06-26 21:07:19][MainProcess][MainThread][INFO ]: (0x7f8648124240) Options: verbose    = True
[2021-06-26 21:07:19][MainProcess][MainThread][INFO ]: (0x7f8648124240) Options: silent     = False
[2021-06-26 21:07:19][MainProcess][MainThread][INFO ]: (0x7f8648124240) Options: stub       = items
[2021-06-26 21:07:19][MainProcess][MainThread][INFO ]: (0x7f8648124240) Options: overwrite  = False
[2021-06-26 21:07:19][MainProcess][MainThread][INFO ]: (0x7f8648124240) Options: backup     = True
[2021-06-26 21:07:19][MainProcess][MainThread][INFO ]: (0x7f8648124240) Options: stock01    = 5
[2021-06-26 21:07:19][MainProcess][MainThread][INFO ]: (0x7f8648124240) Options: stock12    = 5
[2021-06-26 21:07:19][MainProcess][MainThread][INFO ]: (0x7f8648124240) Options: itemids    = ????
[2021-06-26 21:07:19][MainProcess][MainThread][INFO ]: (0x7f8648124240) Options: threads    = 8
[2021-06-26 21:07:19][MainProcess][MainThread][INFO ]: (0x7f8648124240) Options: server     = gilgamesh
[2021-06-26 21:07:19][MainProcess][MainThread][INFO ]: (0x7f8648124240) Options: urls       = ????
[2021-06-26 21:07:19][MainProcess][MainThread][DEBUG]: (0x7f85f80d5278) FFXIAHScrubber: init
[2021-06-26 21:07:19][MainProcess][MainThread][DEBUG]: (0x7f85f80d5278) FFXIAHScrubber: forcing redownload of data
[2021-06-26 21:07:19][MainProcess][MainThread][DEBUG]: (0x7f85f80d5278) FFXIAHScrubber: using passed ids
[2021-06-26 21:07:19][MainProcess][MainThread][INFO ]: (0x7f85f80d5278) FFXIAHScrubber: getting data
[2021-06-26 21:07:19][MainProcess][MainThread][INFO ]: (0x7f85f80d5278) FFXIAHScrubber: executing in parallel with threads=8
[2021-06-26 21:07:19][MainProcess][ExThread_0][DEBUG]: (0x7f85f80d5278) FFXIAHScrubber: open server=gilgamesh (000000/000003,  0.00) http://www.ffxiah.com/item/0
[2021-06-26 21:07:19][MainProcess][ExThread_1][DEBUG]: (0x7f85f80d5278) FFXIAHScrubber: open server=gilgamesh (000001/000003, 33.33) http://www.ffxiah.com/item/1
[2021-06-26 21:07:19][MainProcess][ExThread_2][DEBUG]: (0x7f85f80d5278) FFXIAHScrubber: open server=gilgamesh (000002/000003, 66.67) http://www.ffxiah.com/item/2
[2021-06-26 21:07:20][MainProcess][MainThread][DEBUG]: (0x7f85f80d5278) FFXIAHScrubber: item count = 3
[2021-06-26 21:07:20][MainProcess][MainThread][DEBUG]: (0x7f85f80d5278) FFXIAHScrubber: data count = 3
[2021-06-26 21:07:20][MainProcess][MainThread][DEBUG]: (0x7f85f80e10b8) ItemList: init
[2021-06-26 21:07:20][MainProcess][MainThread][DEBUG]: backup (old): /Users/adam/workspace/pydarkstar/bin/items.csv
[2021-06-26 21:07:20][MainProcess][MainThread][DEBUG]: backup (new): /Users/adam/workspace/pydarkstar/bin/items.csv.2
[2021-06-26 21:07:20][MainProcess][MainThread][INFO ]: (0x7f85f80e10b8) ItemList: overwriting file...
[2021-06-26 21:07:20][MainProcess][MainThread][INFO ]: (0x7f85f80e10b8) ItemList: save /Users/adam/workspace/pydarkstar/bin/items.csv
[2021-06-26 21:07:20][MainProcess][MainThread][INFO ]: exit

To prove that it downloaded different prices:

[AdamGagorik]

I have also updated the docs with an example of choosing the server.

http://adamgagorik.github.io/pydarkstar/generated/scrubbing.html#example

[AdamGagorik]

I will close this since it has been implemented.