Starbound 1.0 support release checklist

[wizzomafizzo]

• [ ] py-starbound support for new database format (SBBF04?)
• [ ] Graphic glitches, countless ones showing up now (see Avian species for example)
• [ ] Crash when opening ship dialog, possibly to do with new character: #271
• [ ] Blueprint dialog crash if null value in list: #260
• [ ] Quests dialog completely broken, need support for new format

Less important:

• [ ] Weapon generation is completely broken (see here for some inspiration)
• [ ] Blueprint list reordering on modification: #249
• [ ] Support for more directives: #261

[wizzomafizzo]

Starfuse by @Qix- is a potential alternative to py-starbound. Will be testing soon

[wizzomafizzo]

Graphic glitches may have something to do with the background of the labels? Need to do some more testing but a lot of stuff magically fixed itself when I randomly changed the background colour

[wizzomafizzo]

@chrmoritz just fyi i think we will end up using starfuse instead of py-starbound (minus the fuse parts). qix is taking a look at the new format in nightly soon, blixt has said he has no time to work on the project at the moment

[Qix-]

Happy to look at it. Any information you have about it will definitely help, too.

[wizzomafizzo]

Shoot man I'll gladly do anything I can to help! I'm not sure what would be helpful though, happy to hear anything you think you need

[wizzomafizzo]

Don't feel like I'm dobbing you in either :P no rush on this stuff

[Qix-]

Well first off, have you tried just adding the SBBF04 string into the array that's checked in the header check for existing libs to see if 'just works'? I'm wondering how much of a change warrants the version number increase or if that's just a way to enforce certain asset files be used with certain versions. I can't imagine the devs changing a simple block format so much between versions, and SBBF03 works just fine with the SBBF02 version.

[wizzomafizzo]

I did but no dice. The new string at least for vanilla assets file is "SBAsset4". It might be as simple as making the string longer than 6 bytes but I wasn't sure if that would break things.

Btw I need to make it work with Python 3 too but I can always fork it and make the changes myself if it can't work for both like py-starbound. As far as I got all I had to do was make the module imports case sensitive and explicitly make some regular strings byte strings. That doesn't really matter for now but just a heads up

Sent from my iPhone

On 19 May 2016, at 1:51 AM, Josh Junon notifications@github.com wrote:

Well first off, have you tried just adding the SBBF04 string into the array that's checked in the header check for existing libs to see if 'just works'? I'm wondering how much of a change warrants the version number increase or if that's just a way to enforce certain asset files be used with certain versions. I can't imagine the devs changing a simple block format so much between versions, and SBBF03 works just fine with the SBBF02 version.

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub

[Qix-]

Gotcha, I can make that a priority for you :)

[rubellyte]

The nightly SBAsset format appears to be very different from the old SBBF03 format. I spent a while trying to figure it out, and this is what I came up with:

• At the beginning of the file, magic bytes "SBAsset6"
• 8 bytes immediately following, offset of file index
• Following that, apparently unmodified files
• At offset specified in beginning of file (end of files), magic bytes "INDEX"
• Byte after INDEX indicating number of metadata entries.
• Metadata entry:
• byte containing length of key name
• key name
• Byte that probably indicates value type (0x02 for float, 0x04 for signed int, 0x05 for string, 0x06 for list, 0x07 for dict)
• Length of value (int values don't get this)
• value
• 1 to 3 bytes of file count (I cannot for the life of me figure out how this is counted, here's some data I collected for anyone who wants to try and solve it)
• File index format:
• Byte with length of path
• Complete file path
• 8 bytes containing offset of file
• 8 bytes containing length of file
• File ends without any sort of marker

I'll update this if I figure out anything else.

[Xirukai]

1.0 is out on the unstable branch. Just giving a friendly headsup, using this program tons ;)

[chrmoritz]

And the release date for stable is the 22 July. 🎉

cc @wizzomafizzo

[rubellyte]

I've got a fork that can read Cheerful player files, but it's mostly useless without something to read SBAsset4, and unfortunately I think that writing something for that is beyond my skills.

[Qix-]

Thanks @medeor413 I can play with it sometime next weekend probably and try it out with starfuse.

[wizzomafizzo]

oops well i guess people are gonna have to chill out and wait a while

@medeor413 are you saying this release changed the player files too? that's a bummer

[rubellyte]

I've already got a fork that works with 1.0 player files, and I started working on something to work with SBAsset, but I'm stuck there.

[wizzomafizzo]

@medeor413 i think the fork looks good i'd be happy to take pull requests if you like

we have to wait for a really nice person to do it :P i'm stuck too

[rubellyte]

I'll put in a PR once I get SBAsset working, since it'll be pretty useless without it.

[rubellyte]

Aaand done. Pull request here.

[chrmoritz]

As Windows nightly build of this PR is available here: https://ci.appveyor.com/project/wizzomafizzo/starcheat/build/master-b59/job/umt2qe3mxck4a9l5/artifacts (if anyone else wants to try it out without having to compile the sources).

[wizzomafizzo]

dude you rock, i'm sure you just made a lot of people happy. i will give it a shot when i get home

have you noticed much difference between this implementation and py-starbound? i just wonder if it got noticeably slower

[rubellyte]

It takes a while for me, but my loading times have always been long due to sheer size of my player files.

[wizzomafizzo]

All good. Hey I'll have to leave this one a bit longer I think, I have no internet atm and can't test anything. If you wanna make changes or have suggestions or whatever don't hold back too, I'm up for whatever

Btw would you be able to upload a screenshot of how it looks now with the UI changes you made? Interested to see them too

[wizzomafizzo]

I wonder if there is a way to speed up the player file loading too? I haven't looked at it in a while but from memory the biggest bottleneck is simply loading up the inventory icons from disk. I'm not sure if it's really avoidable

[kurisubrooks]

Here's how it currently looks. Pillow has some rendering issues, Description and Mouse slots are pending removal, and the hotbar is not currently functional, but the overall function is there.

[kurisubrooks]

I just tested loading Starcheat 0.27 (on macOS) and pending 1.0 release (on Windows 10), but to me 1.0 seems faster.

Overall I don't think there is much of a difference.

[kurisubrooks]

You can see all current outstanding issues here: https://github.com/medeor413/starcheat/issues

[wizzomafizzo]

btw i still have no internet connection at home sorry guys. @chrmoritz if you are comfortable with doing a release at some point please go ahead but i'm ok with waiting until i can test it myself

[wizzomafizzo]

hey could someone give me a run down of where this is at? not sure i'm keeping up. should we be doing a new release?

@chrmoritz could you please help with your git expertise again too i don't know how i would merge the pull request and make dev sync with master

[blixt]

Not sure if you need this anymore, but I've updated py-starbound now to support all the Starbound 1.0 file formats, including write support for SBVJ01 files that's relatively fast (a .player file that's 128 kB writes in 0.029 seconds for me):

import starbound, time
start = time.clock()
with open('SB/storage/player/916d5878483e3a40d10467dc419982c2.player', 'rb') as fh:
  vj = starbound.read_sbvj01(fh)

print('Took %s seconds to read file' % (time.clock() - start,))
print('Player name: %s' % (vj.data['identity']['name'],))

start = time.clock()
with open('SB/storage/player/916d5878483e3a40d10467dc419982c2.player.dupe', 'wb') as fh:
  starbound.write_sbvj01(fh, vj)

print('Took %s seconds to write file' % (time.clock() - start,))

You may also get better speed by using the mmap package.

[blixt]

Oh I misunderstood the bottleneck, for SBAsset6 files you should definitely use mmap for better performance. The export.py script in py-starbound extracts all 44,692 files in packed.pak (892 MB) in 8 seconds for me (compared to Starbound's own asset_unpacker which takes over 10 seconds).

[kurisubrooks]

@medeor413 wanna take a look into this?

[Qix-]

@blixt:

for SBAsset6 files you should definitely use mmap for better performance

I'm sorry I haven't been very active in the SB community lately (full time job took over my life) but my implementation of the SBAsset3 format uses mmap in Python if anyone wants a reference.

[blixt]

@Qix- Ah, great. See below for how I'm currently using mmap, and let me know if there are potential improvements to how to configure mmap!

import mmap, starbound
with open('SB/assets/packed.pak', 'rb') as fh:
  mm = mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ)
  package = starbound.SBAsset6(mm)
  print(package.get('/lighting.config'))

[Qix-]

I'm not sure exactly what Python does under the hood to manage mmap calls, but I know that I gained noticeable performance increases by only mmap'ing the blocks that needed read (the first time they were read, of course) and making sure to map a multiple of mmap.PAGESIZE bytes at a time, starting at offsets of multiples of mmap.PAGESIZE for alignment boosts, which allow reads to be quicker. IIRC the assets file format has a "rounded" block size which helped out with this as well (the pages were an exponent of 2, which helped alignment a bit).

[Malenenshadow]

im having issues with starcheat since today... every time i start it up it stops responding. after five minutes its still like that so im not sure whats wrong since its not giving me an error report

[Malenenshadow]

Update: so it is loading up. but its SUUUUUUUUUUUUUUUPER slow

[blixt]

Do you have the file that's slow? Also I would create a separate issue since it seems to load in the end

El jue., 25 de ago. de 2016 04:29, Malenenshadow notifications@github.com escribió:

Update: so it is loading up. but its SUUUUUUUUUUUUUUUPER slow

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/wizzomafizzo/starcheat/issues/282#issuecomment-242315812, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJrf2UqeuzIfvG78_b8T9wAD4m8Ds9bks5qjVJ9gaJpZM4IfEgO .

[DatDraggy]

Traceback (most recent call last):
  File "C:\Python34\lib\site-packages\cx_Freeze\initscripts\Console.py", line 27, in <module>
  File "build\starcheat.py", line 86, in <module>
  File "build\starcheat.py", line 83, in main
  File "build\gui\mainwindow.py", line 154, in __init__
  File "build\gui\utils.py", line 103, in check_index_valid
  File "build\gui\utils.py", line 64, in build_assets_db
  File "build\assets\core.py", line 127, in find_assets
  File "build\assets\core.py", line 156, in scan_asset_folder
  File "build\starbound\helpers.py", line 213, in open
  File "build\starbound\helpers.py", line 220, in read_stream
  File "build\starbound\btreedb4.py", line 119, in initialize
  File "build\starbound\sbbf02.py", line 89, in initialize
AssertionError: Invalid file format

this is the error. I am trying to convert my old gamesave to the new update. I downloaded this starcheat verison: starcheat-5b0e5ff-win32 I have the newest starbound verison. Is starcheat for 1.0 currently broken?

[kurisubrooks]

Starcheat 1.0 can only open Starbound 1.0 player files. The 2 player file types are incompatible with each other and because of this, trying to load an old file type when Starcheat expects the newer one, will cause it to crash.

Use an older version of Starcheat to access your older player files, export any custom items or weapons you may want to keep, then import then into a new 1.0 player file. This is the best you can do for now.

[DatDraggy]

Thats what I did. I exported it correctly but thats not the problem. Starcheat crashes when I start it. Cant even load a player. Also, my player folders are empty. Both in the old starbound and new.

[kurisubrooks]

That crash log appears to be from an older version of Starcheat. Please use this version, as it is the latest build.

https://ci.appveyor.com/api/buildjobs/mj9h9ri0cxdgfabc/artifacts/dist%2Fstarcheat-9d543cd-win64.zip

[kurisubrooks]

Might I also ask where you found the build that you are currently using? We get so many issues from users using this old build, and it is frankly pissing me off. Was it posted somewhere online?

[DatDraggy]

I followed the steps that is on here. https://github.com/wizzomafizzo/starcheat clicked on TBD - nightly builds available, then the first url, there i clicked the first job name https://ci.appveyor.com/project/wizzomafizzo/starcheat/branch/dev and went to artifacts where i downloaded the zip

[kurisubrooks]

The dev branch is not maintained for 1.0 builds, it's only up-to-date for Starbound Beta, 1.0 is maintained in a separate repository. @wizzomafizzo @chrmoritz Could we please add another link to the README for 1.0 builds?

Thanks for bringing this to our attention

[DatDraggy]

oh okay. Well its something. But is it normal that it takes so long to update the item slots? It's stuck at 0% after I imported the json :/

[kurisubrooks]

Some people are still having trouble with load times, which is why the beta builds haven't hit stable yet. The rest of the 1.0 features are there, but we can't figure out the load times. We've narrowed it down to having to do with HDD speeds, but we're trying to circumvent this.

[DatDraggy]

So I just gotta wait, right?

Well thank you very much for all the help anyways!

[wulfgren]

Is there a link somewhere to the newest version? I've been looking around for it so I can toy add some materials I'm too lazy to farm. I tried the link provided above and it crashes with my all my saves even brand new ones. Feel free to tell me I'm dumb and doing it wrong.

nevermind I found the new version, I appreciate you keeping this up, you're a hero.

[Rensay]

dose the nightly 1.0 currently not work with mod races?