Database download on every startup

[emil-nasso]

I get the dialog that tells me that the database is being updated and downloaded with every startup. Is this intended behavior?

[Bleyddyn]

That is a bug. I think it happens when you have a newer version of the application, but it's trying to download an older version of the database. It will just keep trying.

What's the version of the application, and what values do you have in the two URL fields in the Database pane of the Preferences window?

[emil-nasso]

It's Version 0.3.0a (0.3.0a) And the configuration is http://labs.sixones.com/vitality/database.sql.bz2 and http://labs.sixones.com/vitality/database.xml . Database version says Rubicon 1.0 in the same settings page and language is English. Size is 30.81MB.

[Bleyddyn]

That version should have its own database inside the application which it should use, so I'm not sure why you're seeing that problem. It's possible the unpacking and installation of the internal database is failing in which case it would continue trying to download one.

You can try downloading the newest release v0.3.1b, and/or change your database preferences to point to my dropbox account. That's somewhat iffy because I don't know what kind of bandwidth limits Dropbox has. https://www.dropbox.com/s/y4tcvfz5b2hask4/database.sql.bz2?dl=1 https://www.dropbox.com/s/xioubrs11cvpz0v/database.xml?dl=1

If neither of those options work, you might need to send some debug logs so I can see if there is any useful information.

[mbarthelemy]

I have v0.3.1b which exhibits the behavior as described. I will download and build the source in the near future to see if I can diagnose the problem further.

[Bleyddyn]

It would help if both of you could send me some debug logs.

Open the Console application (in /Applications/Utilities). Make sure the 'All Messages' item is selected in the Log list. Then type 'Vitality' (without the quotes) in the Filter field (top right). Click on one of the error messages, then select all of them and copy/paste them into a message here. If there are a lot of messages, you can just select one session's worth (check the date/time stamps).

I'm fairly sure I've removed most sensitive info from the log messages, but double check to make sure there are no api keys or anything like that. You might also want to delete or xxx out your character names, for example.

Thanks!

[mbarthelemy]

Here are the logs from the console. I write Mac OS X apps for a living so when I get the time (outside of work) I will try to do a bit more diagnosis.

1/8/15 11:57:15.675 AM Vitality[35990]: NSWindow does not support utility styleMask 0x10 1/8/15 11:57:16.191 AM Vitality[35990]: Received Response 1/8/15 11:57:16.191 AM Vitality[35990]: Validating (/Users/xxxx/Library/Application Support/Vitality/database.xml) 1/8/15 11:57:16.195 AM Vitality[35990]: Writing (/Users/xxxx/Library/Application Support/Vitality/database.xml) 1/8/15 11:57:16.196 AM Vitality[35990]: Database current version: 12 1/8/15 11:57:16.198 AM Vitality[35990]: Database current version: 12 1/8/15 11:57:22.846 AM Vitality[35990]: NSWindow does not support utility styleMask 0x10 1/8/15 11:57:22.867 AM Vitality[35990]: SHA1 bz2 hashing failed ('XTEjeP6RIds2i2317c3FIow/JFI=' != 'sWjNIQqBWQIivd+U4vRvMo45hVE=') 1/8/15 11:57:22.961 AM Vitality[35990]: Sparkle configured for http://labs.sixones.com/vitality/appcast3.xml 1/8/15 11:57:22.968 AM Vitality[35990]: Error: cannot find skill 3390 in skill tree! - skipping skill 1/8/15 11:57:22.969 AM Vitality[35990]: Error: cannot find skill 33699 in skill tree! - skipping skill 1/8/15 11:57:22.969 AM Vitality[35990]: Error: cannot find skill 33856 in skill tree! - skipping skill 1/8/15 11:57:22.969 AM Vitality[35990]: Error: cannot find skill 3391 in skill tree! - skipping skill 1/8/15 11:57:22.973 AM Vitality[35990]: Error: cannot find skill 3391 in skill tree! - skipping skill 1/8/15 11:57:22.974 AM Vitality[35990]: Error: cannot find skill 33699 in skill tree! - skipping skill 1/8/15 11:57:22.974 AM Vitality[35990]: Error: cannot find skill 33856 in skill tree! - skipping skill 1/8/15 11:57:22.975 AM Vitality[35990]: Error: cannot find skill 3390 in skill tree! - skipping skill 1/8/15 11:57:22.975 AM Vitality[35990]: Error: cannot find skill 34327 in skill tree! - skipping skill 1/8/15 11:57:22.979 AM Vitality[35990]: Error: cannot find skill 33699 in skill tree! - skipping skill 1/8/15 11:57:22.979 AM Vitality[35990]: Error: cannot find skill 33856 in skill tree! - skipping skill 1/8/15 11:57:22.980 AM Vitality[35990]: Error: cannot find skill 34390 in skill tree! - skipping skill 1/8/15 11:57:22.981 AM Vitality[35990]: Primary character not set, finding first active 1/8/15 11:57:22.984 AM Vitality[35990]: Current character is Xxxxxxxx 1/8/15 11:57:22.985 AM Vitality[35990]: Primary character not set, finding first active 1/8/15 11:57:24.212 AM Vitality[35990]: Error: cannot find skill 3390 in skill tree! - skipping skill 1/8/15 11:57:24.212 AM Vitality[35990]: Error: cannot find skill 33699 in skill tree! - skipping skill 1/8/15 11:57:24.212 AM Vitality[35990]: Error: cannot find skill 33856 in skill tree! - skipping skill 1/8/15 11:57:24.213 AM Vitality[35990]: Error: cannot find skill 3391 in skill tree! - skipping skill 1/8/15 11:57:24.216 AM Vitality[35990]: Error: cannot find skill 3391 in skill tree! - skipping skill 1/8/15 11:57:24.216 AM Vitality[35990]: Error: cannot find skill 33699 in skill tree! - skipping skill 1/8/15 11:57:24.217 AM Vitality[35990]: Error: cannot find skill 33856 in skill tree! - skipping skill 1/8/15 11:57:24.217 AM Vitality[35990]: Error: cannot find skill 3390 in skill tree! - skipping skill 1/8/15 11:57:24.217 AM Vitality[35990]: Error: cannot find skill 34327 in skill tree! - skipping skill 1/8/15 11:57:24.220 AM Vitality[35990]: Error: cannot find skill 33699 in skill tree! - skipping skill 1/8/15 11:57:24.220 AM Vitality[35990]: Error: cannot find skill 33856 in skill tree! - skipping skill 1/8/15 11:57:24.221 AM Vitality[35990]: Error: cannot find skill 34390 in skill tree! - skipping skill 1/8/15 11:57:24.222 AM Vitality[35990]: Finished downloading characters 1/8/15 11:57:24.222 AM Vitality[35990]: Finished batch update operation 1/8/15 11:57:24.224 AM Vitality[35990]: Current character is Xxxxxxxx

[Bleyddyn]

This is the main problem I think:

1/8/15 11:57:22.867 AM Vitality[35990]: SHA1 bz2 hashing failed ('XTEjeP6RIds2i2317c3FIow/JFI=' != 'sWjNIQqBWQIivd+U4vRvMo45hVE=')

I wish I'd been keeping closer track of the database builds and meta info.

[Bleyddyn]

There are some possible failures that won't log any error messages. I've add them, but I'm not sure when I'll be doing a new release.

One thing you could try is to remove these two files, if either one exists:

~/Library/Application Support/Vitality/database.sql.bz2 ~/Library/Application Support/Vitality/database.xml

I checked both the xml and the bz2 file in the download for 0.3.1b and they look correct. The hash 'XTEjeP6RIds2i2317c3FIow/JFI=' is what I'm seeing in both the xml and the bz2 file, both from github and on my machine. I don't know where the bz2 file with this hash 'sWjNIQqBWQIivd+U4vRvMo45hVE=' is coming from.

What do you have in the database pane of the Preferences window in the two URL fields?

[Bleyddyn]

And I found it. The database at http://labs.sixones.com/vitality/database.sql.bz2 has the hash 'sWjNIQqBWQIivd+U4vRvMo45hVE='. So it looks like Vitality is using the xml embedded in the application, but using the compressed database downloaded from sixone's site. But I'm not sure how that's happening, yet.

[mbarthelemy]

One thing you could try is to remove these two files, if either one exists:

Neither of those files exist on my system. Only a database.sqlite (32.3MB) file from June of 2014 and directories for my characters are in that directory.

What do you have in the database pane of the Preferences window in the two URL fields?

http://labs.sixones.com/vitality/database.sql.bz2 http://labs.sixones.com/vitality/database.xml

The database reported in that preferences pane is:

Rubicon 1.0 30.81MB

[mbarthelemy]

It looks like

DBManager.m: -(BOOL) dbVersionCheck:(NSInteger)minVersion;

Never checks if the database shipped with the app is greater than the one that is installed at the default path, which for me is:

/Users/xxxx/Library/Application Support/Vitality/database.sqlite

DBManager.m: -(void) checkForUpdate

XmlFetcher *fetcher = [[XmlFetcher alloc]initWithDelegate:self];
NSString *path = [Config buildPathSingle:DBUPDATE_DEFN];
[fetcher saveXmlDocument:url docName:UPDATE_FILE savePath:path];
[fetcher release];

A XmlFetcher object is alloced and then immediately released. It isn't clear if anything is keeping that object alive to do it's work.

[mbarthelemy]

That object is being kept alive because of a NSURLConnection object and a run loop which is a pretty ugly implied ownership chain. Everything may be leaking.

[mbarthelemy]

It looks like DBManager.m:694 always downloads the database from the same url, which in my case defaults to http://labs.sixones.com/vitality/database.sql.bz2

That value, and the other default URL locations come from MainController's awakeFromNib method which registers them as defaults.

[Bleyddyn]

That value, and the other default URL locations come from MainController's awakeFromNib method which registers them as defaults.

You can override those by putting new URL's in the Database preference pane. However, all of the database and App updating code is pretty messy.

I think using my dropbox URL's is a temporary solution but I'll see about re-writing all of this.

[mbarthelemy]

I think using my dropbox URL's is a temporary solution but I'll see about re-writing all of this.

Yes, that can be a temporary solution so long as you don't use too much bandwidth. Dropbox has a pretty low limit. I haven't noticed where you have published URL's for those though.

I think the best solution to the whole database problem is to build a tool (separate executable) that downloads the original files from CCP and converts them in-place into what Vitality needs. That way you eliminate the need for a third party download site.

[Bleyddyn]

My dropbox URL's are up above in this thread.

These problems are why I switched to including the database inside the Application bundle. I don't think we're ever going to release an updated database without also updating the app, so the original solution never made much sense to me.

[mbarthelemy]

Using those URL's did stop the eternal downloads on startup. I agree that releasing an app with a new database is a better solution to the problem unless the whole process can be completed on the client computer using only downloads from CCP. Many CCP changes will require app changes anyways.

Is there an easy way to turn on non-public skills in Vitality? I have a few skills on my characters which are from ancient times and would love to see them in the lists with everything else. (Most notably the Mobile Factories and Mobile Refining dead-end skills.)

[Bleyddyn]

For non-public skills it would be a matter of removing the "WHERE published = 1" from the select statements in dbscripts/dumprows.sh.

I wouldn't want to see unpublished stuff in Vitality, so a more complete solution would need to include the published field in the exported data and have a preference in Vitality, I think.

on the client computer using only downloads from CCP

The biggest problem I see with this is that the only format CCP officially releases is MSSQL. I have no idea how we would be able to work with that and even if we could, I don't think they have a fixed URL for downloading it. I always get my SDE's from https://www.fuzzwork.co.uk/dump/

[Bleyddyn]

Most of the database update issues should be fixed as of the v0.3.2b release (https://github.com/sixones/vitality/releases/tag/v0.3.2b).