Accurate priceguide

[TeupFleup]

Hi guys, I haven't been following any of these developments to be honest, but was invited by Stellar to share an issue with BrickStock that I've been posting about for some time, and that I consider to be very important as it affects the "lego economy".

BrickStock's priceguide prices do not match Bricklink's. We believe this is due to the fact it assumes everyone is in the US. That means:

• VAT is excluded
• the currency is converted

The latter is no problem for current average prices: you can simply convert them back. But for the sold average (last 6 months sales), this is not possible, because the exchange rates of the date of purchase are used. So, someone may buy in Euro, it gets converted to USD with the exchange rate of that day for the US priceguide, and then in BrickStock a European user converts it back with today's exchange rate, resulting in skewing. And as for the VAT exclusion, well, it's easy to see how that structurally makes prices appear lower than they really are, contributing to the "race to the bottom". People price things at what they think is average, but it's strucutrally slightly below that.

See also what I posted here for a longer explanation: https://www.bricklink.com/message.asp?ID=1245050

I know that the API provides several parameters that should be able to take care of this. So if Brickstore uses the same API, then it should be easy to either translate these parameters to UI options, or to make a default setting based on the user's location.

As I said I haven't been following this so I really don't know where this project stands right now and if this is already being addressed, but anyway, In short, I just think it's important that what Brickstore will provide as the priceguide prices match what you would get on Bricklink. If not, we end up with some very unpleasant subtle-but-long-term price skewing effects that happen without users being aware of it. That is why I think this is a really important issue.

[rgriebl]

I contacted the BL devs about this, but they are currently swamped with work dealing especially with VAT issues in multiple places: they are about to redesign their internal systems, so they are in no position to create a stable API for me at this point in time.

[chiminirc]

I just pulled down build 226. Is that the one? I took all the minifigs and priced them and saved the file before the upgrade. I'm using that to see what values I'm getting in. Trying to get the differences figured out (If I'm using the new method). If I do "download even in already in cache" will that force the new method or is it best to start with an empty cache?

[rgriebl]

Yes - 226 is the correct build. Just force the download -- no need to clear the cache. (If you activate the error log in the View menu, you will see the raw PG numbers printed for each PG download using the new mechanism)

[chiminirc]

I think it is safe to say the new method is being implemented when I force a download. I have 96 lots of minifigs. 28 of them pulled down 0 and the priceguide window on the left is empty.

An example. sw1069. I'm in the US previous method, data pulled today. 6ma N - 9.719 6ma U - 9.021 Current N - 5.94 (verified lowest price in the world) Current U - 6.00 (verified lowest price in the world)

Submitted the set he is in for part out via the web UI. Clicked PRICE on the fig in the part out screen WHOLE PRICE GUIDE matches the values above (expected) MY PRICE GUIDE (all checkboxes checked, 3 decimal) For some reason in the view I ONLY see NEW. Trying to track down if I filtered out USED or if that is expected when I'm pricing a new part for partout. I'm assuming it is wrong to look under "MY PRICE GUIDE" as perhaps it is what I've been selling this item at.

brickstore 6ma N - 10.42 6ma U - 9.35 Current N - 5.94 (Just so happens US is lowest too) Current U - 6.00 (Just so happens US is lowest too)

[chiminirc]

I spot checked some of the ones that got NO price data. Of the 3 I checked, they all appear to have NO data for used sales. My inventory is new parts but it appears if the item doesn't have used data, the new method gets nothing back.

[chiminirc]

Yes - 226 is the correct build. Just force the download -- no need to clear the cache. (If you activate the error log in the View menu, you will see the raw PG numbers printed for each PG download using the new mechanism)

Ok, didn't know about that log view :) So I DO get 2 rows of data for new. Nothing for used so it would appear the code currently won't take the 2 rows it got and display it. It's looking for 4 rows of data.

[chiminirc]

I downloaded my store. 3364 lots. Did a normal price guide update on all of it. Using the defaults of 14 day prices, it felt like it took about twice as long to go through the validation part (progress bar). With the log window open it looks like about 36 items updated their price. No errors. It is also worth mentioning that this progress bar that takes a while seems to only happen when the app first loads. If I load other kits it's always much faster. Not sure if the cache is kept in memory if another document already queried the cache.

I loaded up the Harley Davidson Fatboy and did a forced download of the price data. Parts with no record of used sales caused the priceguide to NULL out. Performance seems about the same. Hard to saw without having both versions running. Nothing significantly faster or slower in this test.

[rgriebl]

Right - I see the same problem when the used data is missing. I'll try to push a fix for that later tonight

[rgriebl]

Build 227 is ready for testing.

[paramecie]

Build 228, yes, that's a closer price (if you didn't change your PG settings on BL).

Tested on: https://www.bricklink.com/catalogPG.asp?P=62462&colorID=47

Same prices, or close due to the rounding:

Great!

...but caution believing the Price Guide is the same for all - see above ;-)

[paramecie]

@rgriebl about rounding:

I prefer to keep 3 decimals in Price Guide widget. The prices in BrickStore documents have 3 decimals also. If BrickStore does better than BrickLink (2 decimals) who cares ;-)

[ZZJHONS]

@paramecie I see 3 decimals on Bricklink too, do you have that activated in your settings?

When I open your link I see 2 decimals too, but when I click View Older version here: https://www.bricklink.com/v2/catalog/catalogitem.page?P=62462#T=P&C=47

I go to the same page but I see 3 decimals, if I open your link again, I keep seeing 3 decimals.

I don't know if it is something about the url part "colorID=47" instead of "ColorID=47"

But focusing on prices, I checked that part with the previous PG change:

New method seems closer yeah (Closer in new, used differs a bit IDK)

[rgriebl]

BrickStore downloads the PG, while NOT being logged in, in USD. These raw USD values are only accurate to 2 decimals (check the error log view for the raw numbers). If you have set your local currency to something other than USD, you will of course see numbers with 3 decimals after conversion.

[paramecie]

I always see 4 decimals in fact, this is how I set it up 15 years ago :-)

It's not the old version, it's the "new" one - see the link on top right of this pic above.

[cwmcd]

@rgriebl i would highly recommend that 3 or even 4 decimals should be a standard. or even more if it's avail.

if you're a store, and you're dealing with bulk lots of hundreds and sometimes thousands of parts where a fraction of a cent matters, it really adds up.

1$ may not seem like a lot but when you've got a bulk lot and the .000X number = 12$, that's quite a bit.

if anything, at least give us the option of selecting how many decimal places we want to view.

on another note, re: pricing guide inaccuracies:

so, i've been doing a bit of testing. i've got bricklink website, brickstock and brickstore.

i'm using set 21326-1 (winnie the pooh) as my test subject.

when i do ctrl+g and force a price download for last 6 months average, i get the following: brickstock: $128.587 brickstore: $65.070 bricklink: (sold) 63.76 (current) 155.86

brickstock shows: qty 31(28) min 100.686 avg 128.587 qavg 127.738 max 150.431

brickstore shows: qty 173(74) min 0 avg 0 qavg 0 max 0 (current inventory also shows q115(75) and all 0's)

bricklink shows: qty 173(74) min .69 avg 63.76 qavg 31.36 max 150.43

doing a part out in brickstock/brickstore: lots 401 items 1316 brickstock: value $308.148 weight: 26.2622 oz brickstore: value $319.820 weight: 27.126 oz (bricklink: value $308.89)

doing a part out of only the sticker sheet item number 21326stk01 brickstock: $3.291 qty 17(15) min 1.116 avg 3.291 qavg 3.330 max 5.325 brickstore: $3.560 qty 17(15) min/avg/qavg/max all 0.000 (also for current inventory) bricklink: $3.29 qty 17(15) min 1.12 avg 3.29 qavg 3.33 max 5.33

so....it's hard to say what i see here. overall i'd say brickstock shows lower and brickstore shows higher. compared to what bricklink shows? well, it's kind of all over the place.

other than the pricing, 2 issues that i seem to note:

1. brickstore has blank fields for all the current and 6mo sales info on the left side price guide pane. not sure if this bug is already listed or not or just on my machine.
2. brickstore requires multiple refreshes to force get data. error log shows 404s. brickstock does not seem to have this issue. i think brickstock takes a bit longer but if i were to have a suspicion, brickstore may need to dial down how fast/frequent they hammer the bricklink servers. that's just a guess and i don't really have anything to base it on other than in past experience with this sort of thing, if the server can't keep up with requests you'll get issues.

in any case, this is NOT to knock on brickstock or brickstore, but just an observation and maybe might be helpful as a 'baseline'.

at this point. as a store owner, all this is very troubling and the lack of focus/communication on/with bricklink's current user base (uk or otherwise) is concerning.

i would not be surprised if the lego group would want to shut out 3rd party apps to force users to use it's existing interface and possibly offer up it's own solution. maybe something to take into consideration. wouldn't be the first corporation to move in that direction.

recommendation? maybe think of working on a modular api system for brickstore so that you can "plug in" or swap api translations for "other" similar marketplaces that have seemed to have more of a 'dev/user' focus than the status quo.

this would also be handy from a troubleshooting standpoint to be able to swap back and forth between apis, and also possibly for an overall plugin system so the community can help expand brickstore's capabilities.

i've always been a big fan of modular systems ;)

just saying....

[rgriebl]

I know the thread is getting very long and the actual facts are hidden between the back and forth, so let me summarize:

1) The APIs that my old BrickStore was using is returning wrong data nowadays (no VAT included) 2) The undocumented, private API that the BrickStock maintainer hacked in is also returning wrong data (no VAT included) 3) None of the APIs are returning exactly what you are seeing when looking at the BL price guide HTML pages. Don't ask me why... 4) I'm in contact with the BrickLink devs about fixing the API or adding new ones, but they are swamped with other work and have no time implementing a sane API for BrickStore to retrieve price guide data in high volume. 5) There's the new JSON based API, but this is (a) slow as heck, (b) returns kilo-bytes of unnecessary data and (c) is limited to 5000 calls per days (with 1 PG request being 4 separate calls to the BL servers). 6) BrickLink's servers are unstable at the moment and they are dropping requests if more than 4 or 5 come in at the same time (the 0 values you are seeing are the result). The industry standard used in browsers is 6 and this has been working fine since 2003. I've dropped the number of connections to 5 in release 2021.4.1 and I will drop it further down to 4 in the next release. (BrickStock is unaffected by this, because it uses a proprietary, batched API) 7) That being said, there is currently only one machine-parseable price guide source that is correct in regards to VAT handling: the in-line PG display when adding items to your inventory. Example URL for Winnie: https://www.bricklink.com/priceGuideSummary.asp?a=S&vcID=1&vatInc=Y&ajView=Y&colorID=0&itemID=21326-1&uncache=1618591116 Please note: these are the exact same numbers you get in BrickStore Also note: there are only 2 decimals to parse - it's either the correct value with 2 decimals or the wrong value (old API) with 4 decimals ... I chose the correct value over precision ;)

So in the end BL is inconsistent with itself: the standalone price guide pages (e.g. https://www.bricklink.com/catalogPG.asp?S=21326-1&ColorID=0 ) are showing different values than the inline price guides when adding items (e.g. https://www.bricklink.com/priceGuideSummary.asp?a=S&vcID=1&vatInc=Y&ajView=Y&colorID=0&itemID=21326-1&uncache=1618591116 ). (you might want to use a US based web-proxy service to see the base USD numbers for the standalone PG to make the number comparable: I used the hidemyass.com / Seattle web-proxy, otherwise I only get EUR values when connecting from Germany)

So which one is showing "the truth"? I don't know...

Find me another usable API that can return correct PG data quickly and in high-volume (thousands) and I'll have a look at adding support.

[rgriebl]

Closed a bit too early - a fix is incoming though :)

[ZZJHONS]

Hello Robert, could you explain what changed with more detail? Thanks

[rgriebl]

Here's the preliminary change-log for the next release: https://github.com/rgriebl/brickstore/blob/main/CHANGELOG.md

Other than that: I received minimal documentation from the BL devs regarding the Affiliate API, so please test the current dev version of BrickStore and report back if the price guide data is wrong.

[ZZJHONS]

For what I see with EU VAT I get averages at 2% to 4% higher than without VAT, and the values are similar to what we got in the previous versions. Seems all good.