search

gsabater / Steam-Backlog

Steam Backlog is a free tool to manage your Steam library
https://steam-backlog.com
19 stars 0 forks source link

Limit of 5000 items in the backlog #61

Closed MakoSipper closed 5 years ago

MakoSipper commented 5 years ago

I know I'm not the rule, but the exception, but I was adding games to my backlog in bulks, and suddenly I noticed after refreshing that many of the game weren't registered.

I currently have exactly 5,000 games in my backlog, across all categories, so I'm assuming that's a cap.

Is it possible to have a larger cap? Otherwise I think I'll be left out =(

gsabater commented 5 years ago

Hello @MakoSipper

We don't have an item limit per se, what we have is a limit on the length stored by the database tied to storage specifications.

What we store in the database is the appid of every game you specify to be in one of your backlog categories. You can see your data if you access https://steam-backlog.com/account/download (the link is available under the account management link in the top menu, use a JSON viewer)

I have looked your library and you have 1710 characters in playing, 3502 characters in completed, 24098 in hold, 2 apps in dropped and 20055 in hidden.

We want to offer a nice product for everyone, you are an exception in terms of owned games, but we don't want to leave users like that out of our service.

I can't give you an estimate because i want to study the implications on increasing database storage size in terms of performance (for everyone), pricing and other things, but we will definitelly take a look at this and i will tell you something next week.

Meanwhile, it would be nice if you could add one game to each backlog category and see if stores in some of them, I guess the issues would be in hold or hidden.

Sorry for any inconvenience and thanks a lot for your feedback, is very valuable for us.

MakoSipper commented 5 years ago

Maybe it will store something, because my 5,000 games are now 4,981.

I've added one game to each category, but sometimes it takes some time before the website excluded some of the games, so I'll wait a bit more.

Is the limit per backlog or per category?

gsabater commented 5 years ago

The limit would be per category

MakoSipper commented 5 years ago

I could some arrangements to avoid filling the "On Hold" category, but the "Hidden" category is a bit harder to work around, I guess.

This is hard situation, I know, I just hope there's a solution even though I'm not qualified to help you come up with one.

gsabater commented 5 years ago

Hello @MakoSipper

We have changed a little our database infraestructure to allow for more storage on any Backlog categories. Now you (and anyone), should be able to set even the double of games to any category.

We have ended paying a little more, but it's worth for not having our users limited. We will be monitoring the libraries for performance on saving and reading, we don't want the service to be slow.

And please, if you encounter any problem that might have been caused for this change, please, reopen the issue and tell us any detail you can.

Thanks

Fixed in October release. Complete changelog in https://steam-backlog.com/changelog

MakoSipper commented 5 years ago

It's still not working for me. I've emptied cache and forced reload, not sure if I should do anything else.

gsabater commented 5 years ago

Hello @MakoSipper

I will need you to provide all the feedback you can. Open the developer tools in Chrome Menu > Tools > Developer Tools.

With the developer tools open, go to the network tab, and try to add a game to your backlog. If there is any error, you will see an entry in red, do some screenshots and attach them here.

Also, tel me how many games are you trying to add, to which category, and how many games where on that category before you was trying to add them.

Also, you could try to remove a game from that category, without adding a new one. Lets see if that triggers an error as well. I'm not having issues, but maybe you have something in your library that we need to spot.

Thanks

MakoSipper commented 5 years ago

No error seen. Let me be more clear:

The games seem to go into the right category, but when I reload the page, it's like nothing happened.

I still managed to add a few more games in batches of 2, 3, and 1. Now I've reached a point when I can't add any more games to Hidden. Tried different games, too.

I think these lines showed up for me when I tried adding Space Hulk to my hidden games: chrome_2018-10-10_11-56-24

On console, there's a single warning on "Steam Backlog getDB: Fetching apps data"

I hope it helps. If not, maybe I can still help you further.

MakoSipper commented 5 years ago

I've also recently found out that I cannot add games to ANY category, even "Dropped", which has only 2 games for me right now.

I'm thinking now the limit is just for the whole backlog, and not for each category.

gsabater commented 5 years ago

@MakoSipper Thanks for your feedback. I have been looking at your library in our database no errors are fired when you save the backlog. But you are right, at this moment, you only have 2 games in dropped.

I have installed an error catcher to better understand what is happening, and I also have setup a development environment where i can have your library and I will try to catch your error there.

I'm sorry this is taking so long.

MakoSipper commented 5 years ago

No worries, just tell me if you need any further help.

gsabater commented 5 years ago

@MakoSipper I have found your problem, and indeed you were right, having more than 5000 games across all categories caused an exception.

Results that we are doing things wrong, and our servers don't handle requests with more than 5000 requests on it. The solution to this is change the way the data is sent to the server before being stored.

This will take some time as we will have to refactor a couple of functions in the system. I will keep you informed, and thanks a lot for the feedback.

gsabater commented 5 years ago

Hello @MakoSipper

To address this issue, we need to:

This is not a very big deal, but it is a breaking change, and i'm afraid we might have problems with users' browser cache.

That is why we are rolling today a change to inform users when they are having a wrong (Cached) version of the assets.

This weekend we will migrate all user libraries to a new format, and we will change how the client and the server communicates.

Again I apologize for the delay, and i know that all this is not directly related to your issue, but i want to inform what is going on

MakoSipper commented 5 years ago

I'm not sure I understand everything, but I'll pay close attention to the future updates. Thank you for your hard work!

gsabater commented 5 years ago

Hello @MakoSipper

We have been making the changes we told you. Can you please test saving games now? You should be able to do it fast and without problems.

Please, be sure you don't have any browser caché, we have reset the assets as well.

Thanks

MakoSipper commented 5 years ago

Seems like all my hidden games have been unhidden.

Is that expected? It will take a while until I can check if I can go over the 5k mark.

MakoSipper commented 5 years ago

I can't add anything to my "Hidden" category anymore, either.

gsabater commented 5 years ago

Seems like all my hidden games have been unhidden.

Is that expected?

Sorry to hear that. Indeed, your hidden games have been unhidden because we lost the data on the migration. I have a backup of the database on my office, and I can restore your hidden games, but will be on monday.

I can't add anything to my "Hidden" category anymore, either.

You can, but only the last added games will be saved, its a bug we are going to solve right now

MakoSipper commented 5 years ago

I tried adding some to Hidden yesterday, but none was saved. I could add games to other categories like "Dropped", though.

MakoSipper commented 5 years ago

You can, but only the last added games will be saved, its a bug we are going to solve right now

Maybe I couldn't because I was trying to re-add games that were supposed to be in Hidden already?

What do you know, now I'm able to add games to Hidden again. Maybe I'll add them manually if I'm bored, otherwise I'll wait until Monday. ;)

MakoSipper commented 5 years ago

Ok, re-added hidden games manually, and had no trouble going over the 5k limit now ;)

gsabater commented 5 years ago

Hello @MakoSipper

First, i'm very sorry we lost your hidden games. You had a lot and we will learn from this mistake, we will be very careful to not lose user data again. We have backups for data, but unfortunately they are on my office.

Second, you have reached the limit of 5000 games and everything is working as intended, right? I will close the issue again, and if you have any problem related to this, comment again and we will take a look at it again.

Thanks a lot for your feedback once again. Half the changelog for this month is thanks to you.

Fixed in October release. Complete changelog in https://steam-backlog.com/changelog