Enhancement: Include URLs in csv downloads

[tetryon]

The web page version of the extension has links, but when downloading those links are stripped. It would be great to have an option to also include the links to URLs in the csv, instead of just the plain text.

Thanks for the consideration :)

[philipmulcahy]

I've not looked at this yet, but it sounds like a schoolboy error by me from a couple of years back. Hopefully it won't be too tricky to fix.

[philipmulcahy]

Ah - I think we might be thinking at cross-purposes. Links that show the URL string in the non CSV report are reported faithfully in the CSV (all the ones I've spot checked), but things that have different display text from the URL, like the order id (for example) are not links in the CSV. This was deliberate - I didn't want to pollute the CSV with extraneous information, making it harder to use - but on reflection, I can see why you'd want the ASIN (for example) in the CSV. I don't see much advantage in linkifying the order ID, because generating the link in a spreadsheet is trivial. Which specific field(s) are you after? Maybe we can add a new column or two to the table?

[tetryon]

Thanks for considering this.

Field 1 - The order number would link to the order in Amazon. Field 2 - The product description would link to the public product page

I do Amazon Vine testing, so I would like to be able to download a CSV, pull it into something like Excel, and have clickable links, and then I would add my own columns after that I side the spreadsheet. I have to review each product, so being able to get straight to the order or product page from a spreadsheet is a big benefit. This gives a one stop shop to oversee my orders, reviews, etc.

I would suggest an option to toggle the feature on/off since not all may want that, and if it makes more sense to add two URL columns then that would also be appreciated.

I can do a mock-up of what the CSV would look like if that would help.

Cheers, AJ.

On Thu, Mar 30, 2023, 19:53 Philip Mulcahy @.***> wrote:

Ah - I think we might be thinking at cross-purposes. Links that show the URL string in the non CSV report are reported faithfully in the CSV (all the ones I've spot checked), but things that have different display text from the URL, like the order id (for example) are not links in the CSV. This was deliberate - I didn't want to pollute the CSV with extraneous information, making it harder to use - but on reflection, I can see why you'd want the ASIN (for example) in the CSV. I don't see much advantage in linkifying the order ID, because generating the link in a spreadsheet is trivial. Which specific field(s) are you after? Maybe we can add a new column or two to the table?

— Reply to this email directly, view it on GitHub https://github.com/philipmulcahy/azad/issues/209#issuecomment-1491105498, or unsubscribe https://github.com/notifications/unsubscribe-auth/AC3GE55NDPYH7JP4TQDM3ZDW6YMHBANCNFSM6AAAAAAWMQHWTU . You are receiving this because you authored the thread.Message ID: @.***>

[archetyped]

Would love to see ASIN for items in CSV export. Currently needing to convert the generated HTML table to CSV to get this info.

[ericcorbin]

I just downloaded the extension and donated. The tracking number is not included in the .csv. It instead says the words "tracking link." If the tracking number were included, I could generate a link. Thanks!

[philipmulcahy]

Is it being lost when being converted to csv or before: e.g. is the link good in the rendered table?Sent from my phoneOn 8 Dec 2023, at 23:39, ericcorbin @.***> wrote: I just downloaded the extension and donated. The tracking number is not included in the .csv. It instead says the words "tracking link." If the tracking number were included, I could generate a link. Thanks!

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[ericcorbin]

The link is good in the rendered table, but when converted to csv becomes just the words "tracking link" with no URL. However, after posting my original comment, I discovered that the link is to a page which tracks the shipment from the Order ID, rather than from the tracking number. What I'm hoping to have is the actual tracking number that is printed on the shipping label included in the csv. That would enable me to know what is in a given package before opening it. (Previously, I did this from the order history report directly from Amazon, which included the tracking number.) Thanks!

[philipmulcahy]

I (may) have just finished the big refactor of query management code that's been tying up all my spare time. I've merged it into master, so now I am aiming to chew through a few bugs, and this is the one I want to do first...

[philipmulcahy]

First bit of progress: https://www.mulcahyfamily.org/azad/azad_7a05bb6c.zip has the full tracking link url, which makes it into the csv file. Here's how to use such a private build: https://github.com/philipmulcahy/azad?tab=readme-ov-file#how-to-try-out-an-experimental-version. If you try it out, let me know how you get on (it's built on top of a newly crafted query engine - big change, so reasonably likely to have some rough edges).

[archetyped]

@philipmulcahy does that experimental version include the shipment tracking number in the CSV?

[ericcorbin]

Thanks for working on this! I just downloaded from the link given, and followed all of the instructions. Unfortunately, I'm still seeing just the text "tracking link" without a URL in the csv. I'm wondering if the zip file has the correct version, as I noticed that it has the same version number (1.9.34) as the other extension (which is disabled).

[philipmulcahy]

The same version number is expected.Did you clear your cache?Sent from my phoneOn 11 Dec 2023, at 03:31, ericcorbin @.***> wrote: Thanks for working on this! I just downloaded from the link given, and followed all of the instructions. Unfortunately, I'm still seeing just the text "tracking link" without a URL in the csv. I'm wondering if the zip file has the correct version, as I noticed that it has the same version number (1.9.34) as the other extension (which is disabled).

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[ericcorbin]

I cleared the cache multiple times. Still getting "tracking link" without a URL.

[philipmulcahy]

That is very strange - I turned off my development copy, unzipped 7a05bb6c and did the "load unpacked extension" dance... here's what I got (after remembering that the tracking link column is only shown for the items view and then only after logging into the extpay dialog):

Before last night's fix, the https://${site_name) was missing from the link, so it'd work from the in-browser table, but not from elsewhere, as the table thinks it's in the amazon site.

Next hypothesis is that you didn't turn off the chrome webstore installed version:

reached from the jigsaw puzzle piece button OR chrome://extensions

[ericcorbin]

OK, I did just test in items view, and it works there. However, the tracking column IS there for the order view, and that's where I'm getting the "tracking link" text. Also, did you see the note about wanting to get the actual tracking number (above)? Thanks!

[philipmulcahy]

Tracking ID (they are not usually pure numbers) is on the way (I got it mostly working last night for my account). I need to look at performance: if I make it too much slower for everyone then the pitchforks will come out, and the workaround (another checkbox) is more work and UI complexity.Let me look at the order level csv behaviour, but given that there can be multiple shipments, it is going to be messy whatever we do.Sent from my phoneOn 11 Dec 2023, at 21:52, ericcorbin @.***> wrote: OK, I did just test in items view, and it works there. However, the tracking column IS there for the order view, and that's where I'm getting the "tracking link" text. Also, did you see the note about wanting to get the actual tracking number (above)? Thanks!

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[philipmulcahy]

https://www.mulcahyfamily.org/azad/azad_76c6e9e0.zip does this:

...in the items view. The tracking id field should make it into the csv without molestation. For my recent orders (the one with tracking IDs), there are three different patterns, from two delivery partners, and they all seem to work when pasted into the relevant tracking sites. There's a fair chance this'll just work on your shipment info as well, because I've used a class name to find the data rather than a human visible string.

Caution: there is some strangeness going on that causes some subsequent scrapes to fail (empty). The work around is to clear the cache AND reload the amazon tab before re-trying. I will work on fixing this before considering releasing.

I am losing enthusiasm for reprocessing (for csv) the links that are embedded in cells with lists in them in the order list view, for three reasons: 1) it'll be super ugly code, that's hard to validate, and keep validated 2) the items view is better for this use-case (as I understand it). 3) the users of the CSV would have to machine parse each csv cell, which is potentially a list of shipments, each with several fields.

If someone actually ends up getting the csv.order.shipments field in the parsed because they need it so much, then I'll suck up the csv translation, but the embedded list is already a bit of a UI war crime. I am also struggling to be enthusiastic about expanding out the tracking url in the orders.shipment in-browser cell, because it'll make the cell (with all of its other info) even more unreadable, particularly if there are multiple shipments per order.

I have two requests: I) testing of the new build and feedback thereof: Does it get your tracking ids? Do they work? (see the caution above). II) thoughts on whether a third view type (shipments) might make all this agonizing go away. Obvious columns to include would be:

• order_id
• items list
• tracking link (in its full glory - no translation needed to get it in the csv
• tracking id
• status
• transaction

[ericcorbin]

Thanks so much for your work on this! It's looking good. Here's what I found, though. If an order was divided into multiple shipments, the tracking ID returned for all of the items is the first shipment only. All other orders are reporting correct tracking IDs. In my opinion, putting the tracking ID and perhaps tracking URL (for me, this is not necessary, as all info can be gleaned from the URL on the order ID) only in the items view makes sense. I don't personally see a need for a separate shipments view. For my use case, I'd love to see the "to" field included in the orders view. If possible, I'd also like the tax per item in the orders view.

Thanks, again!

[philipmulcahy]

"If an order was divided into multiple shipments, the tracking ID returned for all of the items is the first shipment only.” - not good - let me investigate. Can you please send me a json dump file for one the orders you’re seeing it happen on -> @. @.>, with #209 in the subject line so I can find it?

Thanks,

Philip

On 13 Dec 2023, at 14:10, ericcorbin @.***> wrote:

If an order was divided into multiple shipments, the tracking ID returned for all of the items is the first shipment only.

[archetyped]

• Shipment tracking ID in Orders view: I don't think it's necessary, especially if it may contain multiple shipments, as Items view with tracking ID would contain more concise details.
• Separate Shipments view: Seems superfluous. I think the Items view with shipment tracking ID is enough.

[ericcorbin]

"If an order was divided into multiple shipments, the tracking ID returned for all of the items is the first shipment only.” - not good - let me investigate. Can you please send me a json dump file for one the orders you’re seeing it happen on -> @. @.>, with #209 in the subject line so I can find it? Thanks, Philip

Sorry, I don't know where to find the json dump file. Can you point me in the right direction? Also, the email address was obscured in your message. Thanks, Eric

[philipmulcahy]

https://github.com/philipmulcahy/azad?tab=readme-ov-file#how-to-save-and-send-an-order-debug-json-file - it packages up the html data from the pages that are scraped to get that specific order (one of them might contain basic info about up to 9 other orders)

[ericcorbin]

Got it. I've sent the email with the json file. Thanks!

[philipmulcahy]

I'm going to work the getting stuck bug tomorrow and then come back to this, because folks are going to get very annoyed testing this one until I fix the bug. I fear that I'm also going to have to ask you for another json dump when I'm done, because the one you sent me will not have the shipping info - the dump code doesn't know about shipping yet - sorry!

[philipmulcahy]

https://www.mulcahyfamily.org/azad/azad_733fee68.zip was harder work than I expected. It has: 1) no-known "getting stuck" or "cache poisoning" behaviours, though only I've played with it so far. 2) ability to include shipment debug data into debug dump files

@ericcorbin if you wouldn't mind repeating your dump file creation and sending, I'd be grateful. (unless the problem you reported about "painting all the items in an order with the same shipment, when they were in different shipments" has magically resolved itself, in which case just tell me - I figure you won't judge me too harshly for living in hope.)

[ericcorbin]

Thanks for your work on this! Unfortunately, the problem with multiple packages is not (magically) resolved. :-( I sent an email with the new dump file. Thanks!

[philipmulcahy]

1. Good news: I've found a reason why it wasn't working (thanks to your most recent order dump file).
2. Bad news: That reason prevented the dump function from capturing all of the tracking pages.
3. Good news: I've fixed that problem (seems .com shipping layout looks different from .co.uk).
4. Bad news: There might be more bugs that I cannot yet detect because the dump file is not yet complete.
5. Good news: https://www.mulcahyfamily.org/azad/azad_d259caf5.zip

@ericcorbin Would you mind getting me another dump file using the new build? (sorry this is such a drag!)

Yours,

Philip

[ericcorbin]

Again, thanks so much for all of your work on this! I've downloaded the new version and installed it. Unfortunately, it's stalling out on me before loading the order table. I get the orange bar at the top going across, and then when it goes all the way across it just stops there and doesn't load the table. Any thoughts/suggestions?

[philipmulcahy]

Thanks for trying @ericcorbin! I spent 30 minutes playing with d259caf5 when I finished work this evening, and it played ball for me, but my

Do you know how to capture a console log? In this case we might need the full verbose one, which means doing this before pressing the year button:

Also, I suggest clearing the cache before doing your first run: while it's supposed to clear the cache when you install a new version (to prevent new versions being confused by old cache entries that might be in a different format), these test builds all manifest as the same version as the one in the web store.

[ericcorbin]

I've emailed the console log. Thanks!

[philipmulcahy]

@ericcorbin ++ and I have been conversing directly.

Latest and greatest: https://www.mulcahyfamily.org/azad/azad_66970643.zip (but not yet proven to be actually great)

For anyone incensed by stuck scrapes, it has a fix for a bug I found that led to unparseable orders killing the whole scrape. It was caused by a big refactor in how all of the requests to Amazon are tracked during a scrape - the refactor enables much easier debugging, but almost inevitably one or two bugs are going to have crept in that are not tickled by my test pack or personal orders. Luckily I'm finding them much easier to find and fix than their predecessors a couple of years back.

[philipmulcahy]

https://www.mulcahyfamily.org/azad/azad_535ab284.zip

Massive facepalm on my behalf: The bug was caused by the absence of a single full stop in an xpath: bad: "//a[contains(@href, '/progress-tracker/')]", good: ".//a[contains(@href, '/progress-tracker/')]"

@ericcorbin Thank you for sticking with me! Please let me know how you get on...

[philipmulcahy]

While waiting for confirmation from Eric, I'm going to work on the original reason for opening this ticket:

Some linkified fields that are useful in the in-browser table, are pretty useless in the csv version of the table. This is because (for example), the "order id" field in the browser table is a link to the order, but displays just the order id. Some csv users are fine with this, they know how to generate a url from the order id, but some do not.

Here are the fields I'm going after: column name	tables	current behaviour	proposed behaviour
order_id.	both	shows order id in both web and csv. Web links to order. No link in csv	additional column ("order url") in csv, with url
description	items	shows item description in both web and csv. Web links to item	additional column ("item url") in csv, with url

Q: What about the items field in the order table? A: I'm struggling to see many users managing to split up multiple links in the same cell, and they should be using the items table anyway.

[ericcorbin]

This version works great for me! Thanks so much! Great job!

[philipmulcahy]

Thanks @ericcorbin!

https://www.mulcahyfamily.org/azad/azad_89c21b04.zip seems to work for the new fields seems to work for those new fields (which are visible only in the csv downloads) I proposed in https://github.com/philipmulcahy/azad/issues/209#issuecomment-1868576286

@tetryon, @archetyped would you mind kicking the tyres on azad_89c21b04.zip? - You may find how-to-try-out-an-experimental-version useful. Sorry it's taken so long and been so noisy!

[philipmulcahy]

pushing in v1.10.0: should be with y'all inside of 48h (google code review permitting)