Version 0.2.8, CX awards problems

[Timothy0927]

Dear jd20, Thanks for what you have done, this is definitely a super fantastic way to search for awards tickets. I have been using your first version several times, and totally forget to renew and update. In recent days I eventually upgrade the fligtplan-tool to v.0.2.8, and I immediately encounter some problems, and I'll mark below could you please take a look and tell me if I am wrong in any steps?

A. My system macOS Mojave 10.14 node --version v11.0.0 yarn --version 1.10.1 flightplan-tool 0.2.8 p.s. I run "yarn global remove flightplan-tool" + "yarn cache clean" + delete my old flightplan folder(which contained config, data, db files) then "yarn global add flightplan-tool" to make sure I have fresh, new, latest flightplan-tool 0.2.8

B. Steps to set up

1. Run "mkdir flightplan" to create a folder under my user folder
2. Run "cd flightplan" to get into my flightplan folder. Problem 1: But I can't find the folder "flightplan" just like before, although I can get in by Terminal
3. Run "flightplan search" and follow the instructions which lead me to create the file: config/accounts.txt
4. Run "nano config/accounts.txt" and add my accounts

C. Steps to search

1. Run "flightplan search -w CX -p -f HKG -t DOH -o -c business -s 2019-06-25 -e 2019-06-30 -q 2 -h"

2. Problem 2: Unexpected error occurred while searching. Error: Node is either not visible or not an HTMLElement, but the searching kept on, so I eventually got the data between 2019-06-26 to 2019-06-30

   

3. Run "Open data", then I can finally see the folder "flightplan" with sub-folders and I can see the raw screenshots on folder "data", Problems 3 : but if I forget to run "Open data" in next search, I won't see my next raw screenshots.

   

D. Steps to convert to database

1. Run "flightplan parse", the result showed totally different than the first version of flightplan-tool Problem 4 :No matter how many awards seats available, the total rewards found always be 0(zero). And do I still have to run flightplan parse in version 0.2.8?

   

2. Run "sqlite3 db/database.sqlite3"

3. Run ".table"

4. Run "select * from requests segments;"

5. Run "select * from awards; "

6. Run ".quit"

E. Visulization

1. Run "flightplan stats" to see how many awards are there, and Problem 5: the awards results are much more than the raw screenshot we saw

   

2. Run "flightplan server" & "flightplan client"

3. Problem6 : I got plenty of awards tickets as shown, but most of them are not available after manual searching and calling to customer service

real screenshots on the same date

Thanks for reading this long post, let's make all the problems list down

1. Is it normal that the folder "flightplan" become invisible?
2. How can I deal with the error: Error: Node is either not visible or not an HTMLElement
3. The raw screenshots didn't show up if I don't run "open data"
4. Do I still have to run "flightplan parse" and are there any mistakes for my sqlite3. steps? 5+6: The awards results did not reflect the true avalibility.

[jd20]

1. Regarding that you don't see the folder in Finder, that must be a Finder issue... Flightplan's not even creating the folder in this case, mkdir is. I'm on Mojave as well, if I run mkdir flightplan directly from my home folder, it pops up immediately in Finder (I might have slightly different Finder settings than you though). Are you sure you're looking at the home folder in Finder, not somewhere else? Try open ~ from Finder, maybe post a screenshot of what Finder shows.

2. How often are you getting that error "Node is either not visible or not an HTMLElement"? If it happens rarely, it's probably OK, sometimes the webpage responds too slowly, and actions get messed up, but if you re-run the search command it will be OK. If it's happening frequently, I can try adjusting the timing to slow it down some more, that may help (in this case, it's when it's manipulating the calendar to select the proper date).

3. Not sure I understand... open data is just telling Finder to open the directory that has the screenshots. You can also get there directly just using Finder, but you need to first fix your problem in 1. above.

4. No, you no longer need to run parse as a separate step anymore with v0.2+ 👍 So it's working correctly.

[Timothy0927]

Thanks for your instant reply! I am so sorry that I know nothing about any command or software development, so please forgive my stupid questions.

Problem 1, 3, 4 solved after restarting my computer. I saw some bugs Mojave, including the crushing of Finder, or missing screenshots. And now it's all OK after restarting the computer.

Problem 2: Yes, I got that error every time after the first date search

Problem 5-6 are the most disturbing, the results are unmatched to reality.

1. This time I searched "flightplan search -w CX -p -f DOH -t HKG -o -c business -s 2019-06-26 -e 2019-06-30 -q 2 -h"
2. And I got plenty of awards tickets which don't exist in fact.
3. This is the real situation, basically no awards tickets

Could you test this part ? Thank you very much!

[jd20]

Yes, thanks for the detailed report, I will investigate further.

[jd20]

Hi @Timothy0927, are you able to check this again? I believe it is fixed now with 0.3.13.

[Timothy0927]

Hi @jd20

Thank you very much, it's amazing that you instantly debug and updated the new version.

1. Still existed problem : Error: Node is either not visible or not an HTMLElement

   

2. Problem solved: Now the searching results reflect the real condition

This is searching result

This is real condition

The "Miles need" now become "??"

I'll try more to find out whether there is any bug. Thank you :)

[Timothy0927]

By the way, do I have to use "--force" to make sure the searching results are up-to-date?

For examples, if I would love to know

"-w CX -p -f HKG -t ATH -o -c business -s 2019-06-25 -e 2019-06-30 -q 2 -h"

if I run this one more time, 10 days after my first search. Then the queries would be skipped. How can I make sure to maintain the latest result everytime I search?

Do I have to add "-w CX -p -f HKG -t ATH -o -c business -s 2019-06-25 -e 2019-06-30 -q 2 -h --force"?

Thank you very much!

[jd20]

Yes, if you want to replace old results with new ones, you need to add --force. If you just want to fill in the gaps where you have not searched yet, you don't need --force.

--force runs all queries, no matter how recent the results in the database are. An alternative is to use the cleanup command, to remove all results older than a certain time. For example, flightplan cleanup --maxage P14D should remove all results older than 14 days, then when you run the search command you don't need --force.

I'll have to take a look at the missing mileage cost (the ???), possibly a regression.

[jd20]

Pushed 0.3.15, fixes the mileage costs missing, and fixes some awards missing from higher tiers (Choice and Tailored).

[Timothy0927]

Hi @jd20 After updating to latest version 0.3.45, there are 2 major problems during searching,

1. Unable to login, hence no searching results [CX] info Logging in... [CX] error Unexpected error occurred while searching! Error: Evaluation failed: TypeError: Cannot read property 'value' of null

The screenshot would be just like this

After manual login (not headless mode), we can see the awards found successfully

I have seen someone reported that you do need further "flightplan parse --force"

2. The second problem is the old economy problem

Thank you very much, I am grateful for your help :)

[jd20]

Can you please try again? I think all the CX issues should be fixed in latest version now (0.3.48)