Orbital Market not updating...

[hugoattal]

Yep, I know, since a couple of days, Orbital Market doesn't update any more...

It's because the Unreal marketplace added a Captcha to prevent scrapping.

I have a few ideas to bypass that, but I'm kinda tired of playing cat and mouse. So I'm emailing them to see if they can whitelist the Orbital Market server. After all, it only brings them good stuff since all the traffic is redirected to their marketplace.

I'll let you know about their response.

[hugoattal]

(They refused, I found another way)

[MattL-V]

Is Orbital again not updating? Last listed assets are from 3 days ago

[hugoattal]

@MattL-V The marketplace API has become a bit unstable/unpredictable. To solve this, here's how Orbital Market now works:

• It retrieves all the products 100 by 100, starting by the oldest to the newest
• If at one point, the retrieving fails, it retries up to 5 times with a few seconds pause between each
• If it works at some point, it continues
• If it fails at the 5th attempt, it saves the page it was trying to process
• On the next batch (every 24h), it restarts at the saved page that it fails on the last batch

The last batch stopped at product 42300 (there are 52k products), so hopefully, it should be finished retrieving all the products on the next batch.

[MattL-V]

Considering another 24h passed and the assets haven't been updated, does this mean Orbital failed to retrieve the assets for the last 5 days? So, theoretically, this can fail again and again, and Orbital will never display the latest assets? Maybe it would be better to run more often than only once in 24h?

[hugoattal]

@MattL-V Seems like it stopped at 52100. Maybe the marketplace blocks Orbital market every 100 pages (10k products) 🤔 ?

does this mean Orbital failed to retrieve the assets for the last 5 days

No, it means, that it only updated (around) 10k products per day

So, theoretically, this can fail again and again, and Orbital will never display the latest assets?

Since it saves the progression, Orbital Market will inevitably retrieve the latest assets, likely in the next 24h I guess.

Maybe it would be better to run more often than only once in 24h?

I believe the more often I run it, the more likely the marketplace will block Orbital Market 😕...

[hugoattal]

The fact that I don't put much more effort into solving this is that:

• It would require a fair amount of time that I'm currently lacking
• FAB is likely to be released pretty soon I guess 🤔 (they said 2024)

[hugoattal]

@MattL-V, as you can see, the latest products are now available on Orbital Market 😉

[MattL-V]

Unfortunately, the latest assets are from 9 September. Was Orbital up to date for a few days and now has a hiccup for over a week? Does Orbital retrieve all assets from the oldest one every day, or only the newly released ones?

[hugoattal]

You're right, the last week seems to have failed at every single try... I'll check it out, sorry for this 😕

[hugoattal]

It should be fixed + I manually triggered the update. Everything should be updated.

I will wait for FAB to make any update on the update process though 🤔