[FEATURE REQUEST] Import/export all data

[JustinAiken]

Some way to export a csv/xml/json blob.. either just a file dump, or via a http server that runs while the log is open, would be great.

One could then build a GOG Galaxy plugin for it..

[tallbl0nde]

That's a good idea, I'll definitely look into it eventually! Obviously it would only be an export, there would be no way to import the data as I don't know if messing with the log is even possible nor would I want to...

[sawfear]

An integration for importing all the activity data to GOG Galaxy would be really nice! And there sure would be exported data from NX-Activity-Log very very helpful.

[ioistired]

Import would be really nice as well though. I migrated to a new Switch console and now my only play stats are through the official interface, and NX-Activity-Log only shows play activity since I migrated.

[tallbl0nde]

I've been thinking about it and figured I could probably sort out some sort of import that only works for NX-Activity-Log and doesn't touch the system log. This is mainly because I still don't know where this other file is that stores the play data that wasn't "played" on the console before it was reset/migrated. Let me know what you think!

[ioistired]

According to https://switchbrew.org/wiki/Flash_Filesystem#System_Savegames, the save game for title ID 0x80000000000000F0 (pdm:/) holds the "Play Data log", and 0x8000000000000041 holds "Home menu icondata/lru list for recently played games." Together these make up the play activity and allow the switch to display icons for each of the titles. This could be backed up and restored easily, but they're tied to a user account, so would need to be modified to the current user account.

I'm thinking the UX would be something like:

• (on one device) back up activity log
• (on another account / device) restore activity log
• user picks which backup they want to restore
• user picks which account they want to restore from that backup

[ioistired]

An NX-Activity-Log specific import would be great actually! The system log shows my old stats since they're under the same Nintendo account, so NX-Activity-Log is the only place that's inaccurate for me right now. I'd especially appreciate if the import was merged with the existing stats.

[tallbl0nde]

With regards to those saves, I'm already using 0x80000000000000F0 (pdm:/) for all the 'Recent Activity' stats (it just contains timestamps of launches/etc). This gets wiped with a factory reset (and probably isn't sent to N's servers) so thus there must be another place the total time is stored.

And like you said you'd have to know which ID is associated with each user, which I suppose I could show in the backup stage and it's just needs to be screenshot or written down (or actually now that I think of it I could write the username into the backup).

The main problem with restoring to the system is I'm not sure how safe it would be to directly mess with those saves (I'm assuming inserting icons without doing other stuff probably isn't great) plus I know the pdm:/ save is unwritable while Horizon is running. That's why I feel an "app" backup/restore would be easiest. I know this would kind of constrain those who really care about their play stats to my application, but I just don't think writing to the system saves is a great idea...

And yeah I'd definitely merge the two, the easiest way for me would be to choose an end date when backing it up so I don't need to program any duplicate checks, but I feel that's just lazy so I'll see :P

[ioistired]

Could probably just ignore if the (program_id, timestamp) is identical, as it's extremely likely to be unique.

I know this would kind of constrain those who really care about their play stats to my application

What even would be the purpose of restoring said data other than for the app? Doesn't the built in activity viewer only show data tied to the account?

[ioistired]

If you want, I could work on a JSON/MsgPack based backup/restore implementation.

[tallbl0nde]

Could probably just ignore if the (program_id, timestamp) is identical, as it's extremely likely to be unique.

Yeah I'm dumb, they would be unique!

What even would be the purpose of restoring said data other than for the app? Doesn't the built in activity viewer only show data tied to the account?

You're right, I was thinking of restoring to other accounts but I suppose that's not really needed anyway.

It would be great if you want to work on implementing it, I'm quite busy with other stuff right now so I don't know when I'd get around to this! :)

And on another note I think I know why it doesn't show the correct amount of hours - most likely the user ID changes and since I'm only querying for the present ID the older records are ignored...

[ioistired]

And on another note I think I know why it doesn't show the correct amount of hours - most likely the user ID changes and since I'm only querying for the present ID the older records are ignored...

Not sure which situation you're referring to here. In my case it doesn't show the old hours because they're literally not on my current switch unit at all.

[tallbl0nde]

I was referring to logging in with the same Nintendo Account after a reset/on another Switch, as it shows your previous hours in the User Page. Not sure if that's what you mean here but I know it's an issue with others.

[ioistired]

Regarding import: is there currently a place that the app caches play activity? I dug at the source a bit and didn't find it, but maybe I'm missing something. If there isn't one, that's fine. I'm thinking the import would read all system activity into memory, then read all app activity.

[tallbl0nde]

If you look at PlayData you'll find it caches the parsed play events used for Recent Activity. However, it doesn't make the raw events accessible so you'd need to add a new function which returns them in a vector or similar.

In terms of the "summary" (All Activity) it reads from Horizon - that isn't cached, however it's easy and fast enough to just call the following function when the backup is requested for all title IDs: https://github.com/tallbl0nde/NX-Activity-Log/blob/5e8d8b10b666983dc978156ecb959c4eb5211b7b/source/nx/PlayData.cpp#L364

[ioistired]

What format do you reckon I should use for the internal database? I was thinking a JSON file that looks like this:

{
    "accountID1": [
        { /* play event #1 */ },
        { /* play event #2 */ },
        { /* play event #3 */ }
    ],
    "accountID2": [
        { /* play event #1 */ },
        { /* play event #2 */ },
        { /* play event #3 */ }
    ]
}

These arrays would be sorted by time. This file would be empty in most cases. It would only be filled up after an import.

For export, one or more accounts are selected and a file like that is created. On import, the user chooses which backup to restore, and then can select which account they want to restore. They choose which account they want to restore to. Let's say they pick account ABCDF to restore to account GHJKL. Then internal_database['GHJKL'] is merged by time with backup['ABCDF'] and the resulting internal_database['GHJKL'] is merged with the entries retrieved for GHJKL from HOS.

Subsequently, on startup, if the user selects account GHJKL, then those entries are read from disk again and merged with the entries from HOS in memory.

Does that make sense?

[ioistired]

Hmm… that doesn't account for "All Activity".

[tallbl0nde]

I'm not sure if this is overthinking it, but I know how to get SQLite to compile + run on the Switch, so what would likely be more efficient is storing imported events in a proper database. On an import it gets filled with all the relevant events and then has a user ID associated with them (as you mentioned).

I think a JSON format like that would be great for importing/exporting. I'd rather not have the events sorted though as for example if you had: 26/8/19 1:00pm Launch 26/8/19 1:30pm Suspend 26/8/10 11:00am Resume ... They would be placed in the wrong order (note this is only if the time is changed while a game is running, which I believe some people do). Then when sorted the events may coincide with more events of the same game and the stats would be reported incorrectly. Thus dumping them in the order they're stored is what I'd prefer.

I definitely like how you described the import/export process, that's sounds great! And for all activity there could be a separate array like:

{
....
    "allActivity": {
        "accountID1": {
            "titleID1": {
                 "totalSeconds": 2345,
                 "launches": 42,
                 "firstPlayed": <either date timestamp or seconds here>
            },
            "titleID2": {
                ....
            }
        }
    }
}

and so on... Obviously the account ID may be incorrect when imported so this would also be stored in the database with the new ID.

Let me know what you think! :)

[ioistired]

Using SQLite would be pretty rad. Is it in the dkp repos? Everything else you mentioned seems viable too.

[tallbl0nde]

It's not in the repos, and it's still quite finicky to get working (probably because I haven't configured it properly). Long story short I had to:

• Change a few lines in the source to get it to compile (due to missing headers/libraries in dkp)
• Omit some features using defines (not important features though)
• In my code actually initialize it a certain way (otherwise it fatals)

And finally for some reason it won't create and write to a blank file, so I had to create a database with empty tables on my PC which the Switch then filled 🙃

So yeah it's a bit of a mess, but I've been using it for another homebrew project I've been working on and I've had zero issues since I got it all of the above sorted out! If you're interested in working on it I can provide the details of how to set it all up, otherwise I'm happy to integrate that side of things once you've got the export sorted out.

[ioistired]

Sounds a bit too finicky for my liking. Maybe we could just go with a simple binary serialization library like https://github.com/google/flatbuffers.

[tallbl0nde]

That looks fine. In the end I'm not really fussed how it's done, just as long as it works! :)

[38285760]

@tallbl0nde @iomintz

Hello, is there any news regarding this? Change my switch and before saying goodbye to my old switch I would like to keep the record.

I would really appreciate it

[tallbl0nde]

@tallbl0nde @iomintz

Hello, is there any news regarding this? Change my switch and before saying goodbye to my old switch I would like to keep the record.

I would really appreciate it

I haven't started work on it, I'm not sure if @iomintz has either. I've been working on another project which is why this hasn't been receiving many updates recently. Sorry but I don't intend to work on this for a little while yet... I also don't want to throw something together quickly which may not work as intended either.

[ioistired]

I too have been working on another project :slightly_frowning_face:

[ygordreyer]

@ioistired @tallbl0nde I know this is an old issue... But I'm thinking about implementing an integration of the Activity Log to the Playnite Software. It would be incredible to automatically track the Nintendo Switch Activity through there... it would be a game-changer for me. Have you guys made any progress on that? If not, I'll see how it would be possible to make it.

[tallbl0nde]

Exporting was completed as of a few days ago ;) You can see the dev branch if you're interested in the code, but it essentially dumps all play data to a JSON along with user names, game names and their IDs if that's relevant.

[tallbl0nde]

Added in v1.4.0!

[ioistired]

I don't think this should be closed until it supports All Activity

[tallbl0nde]

@ioistired That's true, but I can't see any feasible way to import All Activity data.

'All Activity' is just another phrase for 'data that can retrieved using easy libnx methods'. 'Recent Activity' similarly means 'data processed from individual events stored on the switch'. In an ideal world, all the data represented in All Activity would be present in Recent Activity, however these events are wiped on a factory reset, whereas All Activity still returns stats from previous data that isn't included in the events.

It's because of this I can't see it working; I can't ever rely on the Recent Activity events to account for similarities/differences in All Activity (in my situation, All Activity includes playtime from 2017 however Recent Activity doesn't show anything behind 2019 as I wiped my switch + restored my user). I just have two sets of statistics with no meaningful correlation between the two in terms of 'overlaps'. There would also be a similar scenario when trying to import different users, etc.

I hope this explains the process and why it's not really possible!

[ioistired]

Wait, so Recent Activity could be the same as All Activity? Like it doesn't just show the past 30 days or whatever? As for All Activity, couldn't the software pull from both an internal database and libnx?

[tallbl0nde]

Recent Activity could be the same as All Activity, if all the play events that are collated into the 'summaries' are stored on the Switch. All Activity essentially reads the same data that the stock User Page does regarding play time, whereas Recent Activity uses my own code to analyse the data recorded on the Switch. All Activity is essentially guaranteed to be accurate as it uses whatever code Nintendo has written to process play time, where as Recent Activity could be error-prone, etc. (which it is as I explained before).

As for All Activity, couldn't the software pull from both an internal database and libnx?

Of course, but how do you combine (e.g.) 112 minutes of playtime (from Nintendo/libnx) and 59 minutes of playtime (from DB)? How do you know which parts overlap (if they do)? All I have are two sets of numbers, I'd like to use the Recent Activity data to determine how they overlap but if someone has transferred/restored their user, or are missing the local play events for whatever reason it's impossible to do so. It's because of this it felt easier to leave out the import otherwise it'd confuse users and make it more complex than what it needs to be.