Crash when attempting to load games.

[JonathanLochridge]

Status: Crash on load but not save.

After downloading this program and installing PIL. I was able to get it run smoothly. However, whenever I attempt to save or load a game it crashes. I was able to get it to work without this problem three years ago on a different computer from the Sourceforge link. I am unsure of what is different between that version and the one I have today that is causing the issue. In my folder hunting I have not found been able to find the crash dump yet to try to figure out the issue.

Also I found it does not do anything when I click the game settings on the main menu.

[lassefolkersen]

I tested and saving crashes for me as well. As you can see the last active code commit was 8 years ago. I suspect something changed with the pickle module in python. Or else that save-function just always had some problems.

I'm sorry but I don't have time to maintain this anymore, so I'm not sure how to fix.

[JonathanLochridge]

I have been poking around and I think that the code that saves the game is in solarsystem.py But I am not completely sure. If that is correct do you know which file should be logging what happens in that file? I thought that intro.log-psyco and main.log-psyco might have logged what happened when the program crashed but The dates on the log do not match up with any of the times or dates I ran it so perhaps it is unable to save the log data when the program ends. I managed a work around by doing a screen capture of the python terminal as I tried to save it. The error that the console indicated was a IOerror: [Errono 2] No such file or directory: u'savegames\Test for Save 4' It said the it came up on line 349 in save_solar_system while in the solarsystem.py file. Which is file = open(filename, "w")

It looks like it crashes before it tries to cpickle it on line 350. Or to pickle it normally on line 354.

[lassefolkersen]

I don't know. Psyco is just a speed-up add-on, it can be switched off and the game runs the same, just slower. I don't think that's to blame. I'd blame the pickle module (as I did in previous post), but from what you write it sounds more like just an ordinary bug.

Does it change anything if you give savegame names that are only characters (no spaces, no special characters or such)? Alternatively it sounds like the savegames folder is not correctly made out, somehow, that could totally happen. I'd maybe try to create a folder named savegames in the root game folder.

Sorry if it sounds like I'm rambling, but I didn't look at this code for more than 8 years... totally forgot a ton about it

[JonathanLochridge]

Thank you I will try to change the way I make the savegame names and add a savegames folder tomorrow and see how it goes.

Ok. I added a "savegames" folder to my local version. It fixed the first problem. But we are still not finished with fixing it. I saw the message "some weird error with cpickle -shown here:" then a bunch of file paths. I also get a RuntimeErrror for maiximum recursion depth exceeded in cmp. I recognize the message as the command in solarsystem.py printed by line 358 if the attempt to run cpickle on line 350 fails due to an error that is not a MemoryError.

Because it is not a memory error the program is not trying to run normal pickle when cpickle fails. I am going to try copy pasting the code to try a normal pickle inside the non MemoryError exception. This is to see if it is just cpickle that is having problems or if there is some issue that stops both kinds of pickling.

Despite crashing, now we get a save file when it crashes. I am unsure if it has everything it needs to be able to be loaded. I tried to load the file and I got further than I did before. When I did have a savegame folder or a save file it would crash as soon as I hit load game. Now it opens a menu to choose from saves. However when I highlight the save and hit okay to load it crashes.

The error for loading is a TypeError. It gets to line 239 of intro.py before crashing. In load_game There is only one command in load_game. It starts up the main loop with an argument for the savegame based on the name of the file of the save.

TypeError: start_loop() got an unexpected keyword argument 'load_previous_game'

The 'load_previous_game' is set equal to the name of the file found at the path in "savegames". However, because 'load_previous_game' is an unexpected keyword I think that the argument might be the wrong data type for the function. I will check up to see if the variable/data types match between what intro.py sends and load_game receives.

I checked and the variable names did not match between the files. I fixed that on my local test version and came across an error in load_solar_system on line 389 of solarsystem.py AttributeError: solarsystem instance has no attribute 'solar_system_object_link'

It looks like the above error is an error within an exception that is thrown if cpickle is unable to load a file. So I think that means that the file that the saving function makes is not currently viable enough for the program to load with it.

So I went back to trying to fix the RuntimeError and I think the problem involves one of the data fields of solarsystem.py. I think the most likely place to have a problem is where the images are stringified between lines 322 and 348 of solarsystem.py. I think that I don't understand how the stringifiication works well enough to debug it. But I might be able to figure out something later. I will create another pull request for my change to intro.py to make it send a variable of the right name to load_game.

[lassefolkersen]

This is excellent. Thanks a lot I'm going to handle in detail and take in pull request as soon as I'm back with my laptop. You really make me want to take up the project again. I remember that the genetic algorithms governing how the "AI" and market mechanisms work, they really could be improved as well. I learned lots of stuff over last 8 years that could be used

[JonathanLochridge]

I would appreciate it if you did take up this project again. I really enjoyed playing this game when I was a teenager. I wish I was actually any good at it though. I usually went bankrupt if I did not increase my starting income by a lot. I also had trouble trying to figure out how to sell labor well when I played as a country.

I think I am probably stuck when it comes to fixing the recursion error bug. I looked up stuff about it and as long as it is not set up in a way that causes an infinite recursion attempt to hit the limit. We could just raise the recursion limit. However, I am unsure of where I should do that if I don't want to mess anything up. I might try and hope for the best tomorrow but I don't feel like doing that kind of debugging today. However, if it turns out that the program has a more serious recursion problem then I have no idea where to look for the problem. It could be any data field connected to solarsystem.py. Because self is what is sent to the pickler in solarsystem.py. Potentially, I could have to go in to every file of the program to look for any places it could be saving something to a data field on solarsystem.py that could be messed up. I am not sure if I can fix this bug. But If I have to I can put a few hours in this next week trying to understand the code better to see if I might be able to understand it well enough. I still have not read through the whole thing so I will schedule some time to do that this next week.

[JonathanLochridge]

Ok, I think I managed to read through all of the code besides the log-psyco stuff. I still don't know how to fix it yet. I also did some reading on how pickling works so I plan on try to read one of the save files to see if I can find something wrong with the file itself. I have a better idea now what the program is actually trying to save now. I am not completely sure about which exact things are supposed to be saved because, I was unable to find a list of every data field attached to solarsystem.py. There are things that I am sure is saved. Such as the list of companies and their firms and bases. As well as the research tree. I don't know if the financial history of firms is saved or not though. Because I found a function that was supposed to delete the oldest financial data for a company when there was enough of it.

Anyways, hopefully I can get a clue as to where the problem is when I look at the pickled save file. If I don't get a clue from that I don't know what else I could try to find the solution.

[lassefolkersen]

You can definitely ignore all the psyco stuff. It's just a speed-up add on. In fact it can be switched off completely, and it'll run just the same. If you still suspect that's a cause then try to switch it off here: https://github.com/lassefolkersen/highfrontier/blob/6bc82e0b869124a2b9b2a9035ae2f2591009ca3b/highfrontier/src/highfrontier/main.py#L94

The issue with the pickling. Yes that sounds right what you write. I definitely think the things needed to save can be slimmed down. I believe it also saves too much stuff. I think that's even alluded to in this comment:

https://github.com/lassefolkersen/highfrontier/blob/6bc82e0b869124a2b9b2a9035ae2f2591009ca3b/highfrontier/src/highfrontier/solarsystem.py#L269

I think steps forward would include to insert a print + stop into the function, then run the game and save, and then inspect the contents carefully to see what else can be thrown out. Basically as you already wrote

On observation I seem to recall is that it always could save right after game start, but the problem arose after some time have passed. Seems consistent with it adding something that can't fit into the pickling. Can you reproduce that observation?

[JonathanLochridge]

On observation I seem to recall is that it always could save right after game start, but the problem arose after some time have passed... Can you reproduce that observation?

I can not reproduce this observation. All but one of my previous tests of saving where done as soon as I created a new game.

I think steps forward would include to insert a print + stop into the function, then run the game and save, and then inspect the contents carefully to see what else can be thrown out. Basically as you already wrote

I added a print + stop to my test version. When I ran it, it printed two things. It printed and A and B are memory addresses that I don't particularly want to type out.

It does not look like this has anything useful to me. It does confirm that some data about companies and some data from solarsystem.py is included. but not which data. I am going to try adding another print statement to print solarsystem.solarsystem and company.company. Hopefully that will cause it to display the contents of the memory location. But, I am not very optimistic about that. When I goggled stuff about it. All I got was people trying and failing to use pointers in python. Because python does not have pointers.

[JonathanLochridge]

I managed to get past the recursion error thing.

However, now I get a RuntimeError. RuntimeError: dictionary changed size during iteration.

[lassefolkersen]

Maybe that helps if you iterate over a sequence, instead of the dictionary.

for i in len(dict):
   dict[i]

Instead of dict.keys()

Or something. I've seen that before, at least, but I'm unsure where exactly in the code it'd be

[lassefolkersen]

Oh, wait - now I'm mixing it up with some R-coding trick. Wrong language, sorry

[JonathanLochridge]

I don't know where in the code it is either. If you know what types of commands might be giving an issue I can search in each of the program files until I find possible problem areas.

[lassefolkersen]

Hmmm, it could be the for loop in the save function:

https://github.com/lassefolkersen/highfrontier/blob/6bc82e0b869124a2b9b2a9035ae2f2591009ca3b/highfrontier/src/highfrontier/solarsystem.py#L324

But unfortunately I think more likely that it is a dictionary loop in the pickle function itself

https://github.com/lassefolkersen/highfrontier/blob/6bc82e0b869124a2b9b2a9035ae2f2591009ca3b/highfrontier/src/highfrontier/solarsystem.py#L350

Sorry I can't be more helpful. I never did fix it myself

[JonathanLochridge]

I think I am going to try to post this problem to stack overflow to see if anyone over there has a solution. I also am going to go ahead and read through the incomplete save file for clues this afternoon. I wish I had a view of the data before and after the pickle so I could compare them to see where the pickle was messing up though.

[JonathanLochridge]

Here is the stack overflow page link: https://stackoverflow.com/questions/57312734/how-to-fix-runtimeerror-dictionary-changed-size-during-iteration-with-python

[JonathanLochridge]

I don't think commit #5 part: ff4b01d, fixes this but I could be wrong, It does seem to mess with some smaller pieces of pickled data which seems to work. But not the whole game/solar system. Just something for individual planets maybe?

Perhaps that suggests that having multiple separate pickled save files that are loaded in sequence rather than trying to load the whole solar system might work, seems a bit more trouble to do though.

I am going to do some testing though, Maybe this does make saving work again.

[JonathanLochridge]

I temporarily won't have access to my windows computer for testing for a couple days, I meant to get the current error uploaded, but it will have to wait till this Saturday at the earliest, possibly next Wednesday even. Unless I manage to get past the proj/linux error on game start before then.

[lassefolkersen]

I put in some work on this, and now saving/loading runs as part of the github actions tests, here, specifically this function. It includes a pretty comphrensive check if the before/after status is the same, and I think all looks good, but I did not test yet.

[JonathanLochridge]

After, testing with the new unmerged PR, it saves without a crash, but still crashes on load.

Error is below:

Traceback (most recent call last): File "/highfrontier-ci_cd_tests/solarsystem.py", line 415, in load_solar_system new_solar_system = pickle.load(file) ^^^^^^^^^^^^^^^^^ EOFError: Ran out of input

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/highfrontier-ci_cd_tests/intro.py", line 396, in introGui = IntroGui() ^^^^^^^^^^ File "/highfrontier-ci_cd_tests/intro.py", line 70, in init self.receive_click(event) File "/highfrontier-ci_cd_tests/intro.py", line 86, in receive_click button.activate(event) File "/highfrontier-ci_cd_tests/gui_components.py", line 499, in activate return_value = self.function(self.label, self.function_parameter) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "highfrontier-ci_cd_tests/intro.py", line 239, in load_game self.main.start_loop(loadPreviousGame = os.path.join("savegames",load_window.selected_name)) File "/highfrontier-ci_cd_tests/main.py", line 43, in start_loop sol.load_solar_system(loadPreviousGame) File "/highfrontier-ci_cd_tests/solarsystem.py", line 418, in load_solar_system self.solar_system_object_link.messages.append(print_dict) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'solarsystem' object has no attribute 'solar_system_object_link'

[lassefolkersen]

I'll put in some more work on this now. Let's see