search

CUTR-at-USF / MUSER

App used in USF research on music therapy. Based on Shuttle Music Player (https://github.com/timusus/Shuttle).
Other
2 stars 1 forks source link

Log basic music player and UI event data to Firebase #17

Open barbeau opened 4 years ago

barbeau commented 4 years ago

Is your feature request related to a problem? Please describe. We need a way to identify users to be able to associate the UUID assigned to the music playing data with a user identify - for example, an email address. This is required to cross-reference the music playing data with other data on the caregiver/receiver collected via other methods (e.g., web survey).

For now, we can target single caregiver / care receiver model.

Then, the app should log usage data using this UUID to Firebase.

EDIT Aug 2020 I'm updating this issue to just focus on logging data to Firebase and will break off the other tasks to separate issues.

Describe the solution you'd like We've implemented something similar in the OneBusAway open-source app: https://github.com/OneBusAway/onebusaway-android/

Example prompt for user registration in the OneBusAway: https://github.com/OneBusAway/onebusaway-android/blob/master/onebusaway-android/src/main/java/org/onebusaway/android/ui/HomeActivity.java#L396

We store email->UUID mapping in Google Sheet, and we store travel behavior info w/ UUID in Firebase. More info in presentation and final report from project:

Player Events that need to be logged:-

UI Events that need to be logged:-

barbeau commented 4 years ago

If anyone wants to download the try out the OneBusAway app to see how regsitration for data collection UI works, you can download it from Google Play here: https://play.google.com/store/apps/details?id=com.joulespersecond.seattlebusbot

Try starting the app, then killing it, and then restarting it - you should be prompted if you want to help improve transit - tap "yes" to continue with enrollment. But don't actually agree to Informed Consent unless you want to enroll in study.

barbeau commented 4 years ago

@BumbleFlash VERY IMPORTANT - if we copy code from OneBusAway into this project, we need to be sure to change the script IDs for the Google App Scripts before running the application, otherwise it may insert bad data into our OBA data sheet.

BumbleFlash commented 4 years ago

I'll keep that in mind before running the app. Thank you for the heads up. @barbeau

barbeau commented 4 years ago

Looks like this is the Android class to extract metadata: https://developer.android.com/reference/android/media/MediaMetadataRetriever

Example usage: https://stackoverflow.com/a/11328221/937715

barbeau commented 4 years ago

Related to https://github.com/CUTR-at-USF/MUSER/issues/4 - Looking at MediaMetadataRetriever, I don't see a constant for ISRC.

BumbleFlash commented 4 years ago

The list of the events that we need to capture are:-

BumbleFlash commented 4 years ago

I tried logging the song ID and this is what it was. 2020-07-14 18:01:02.957 24914-24914/edu.usf.sas.pal.muser D/SongID: 25

I don't understand how Shuttle uses a 2 digit number to uniquely identify a song unless the ID is generated locally. I'm assuming if I have two copies of the same song, there will be two different IDs.

barbeau commented 4 years ago

Yes, I assume the ID is probably just a local primary key in that database table.

barbeau commented 4 years ago

If the built-in Android MediaMetadataRetriever doesn't work, we could also try 3rd party libraries like: https://github.com/mpatric/mp3agic

barbeau commented 4 years ago

However, the above 3rd party library may be only MP3-specific - it may not support as may formats as the more generic MediaMetadataRetriever.

BumbleFlash commented 4 years ago

Thank you for sharing the links. We can effectively push the song's metadata to Firebase using these libraries however I checked out another couple of libraries that access metadata, unfortunately, none of them seem to extract the ISRC.

BumbleFlash commented 4 years ago

I don't understand why there isn't a single library that we can use to read ISRC codes. Searching it using different keywords would only lead me to embedding codes as opposed to reading them. Moreover, the libraries that read metadata do not provide a function to read ISRC codes. Is there some kind of legal restriction on reading the ISRC codes? I'm so confused.

These are the libraries that I looked into:-

barbeau commented 4 years ago

For our purposes now let's just focus on extracting the metadata we can with MediaMetadataRetriever and see what's typically populated in some example songs. We can then look at https://musicbrainz.org/doc/MusicBrainz_API or another API to see if we can look up the ISRC from that info, most likely in postprocessing after the info is extracted from Firebase.

barbeau commented 4 years ago

Also, could you please update your comment above to include a list of the libraries you've looked at so far, so we have a record?

BumbleFlash commented 4 years ago

While a track is being played, the Song object stores the metadata of the same. These are the values that the object stores:-

Should we consider pushing the entire Song object to Firebase?

barbeau commented 4 years ago

Yes, we should create a POJO (or Kotlin) object, populate as many fields as we can with song metdata, and then just push that container object into Firebase (for now - we can optimize later).

Song is a Shuttle class?

If so, please see where it's populated, and if it's using MediaMetadataRetriever, and post a link to that code in this issue. We may need to expand the code using MediaMetadataRetriever to populate more fields, if we're interested in them. Please take a look at all the constants in https://developer.android.com/reference/android/media/MediaMetadataRetriever and see if there is anything not retrieved by Shuttle that we might find useful.

If Song qualifies as a POJO (there isn't any other implementation logic), then yes, we can push that object to Firebase.

BumbleFlash commented 4 years ago

@barbeau I have put together information about Song.class and MediaMetaDataRetriever.class listing the properties that are common and exclusive to one another and shared it with you. Here's the link to the document. Please feel free to let me know if you're unable to access it.

barbeau commented 4 years ago

Link to Kotlin model class example: https://github.com/OneBusAway/onebusaway-android/blob/master/onebusaway-android/src/main/java/org/onebusaway/android/nav/model/Position.kt

barbeau commented 4 years ago

To break up PRs in implementation, I've changed this issue to focus on logging usage data to Firebase, and I've opened these two issues to focus on the other parts of the workflow:

BumbleFlash commented 4 years ago

Thank you, Sean. Also, here are the details about the new mediainfo that could potentially be referred to for extracting metadata that couldn't be done by using the traditional content provider.

Please check this potential tool to get mediainfo. This is the GitHub repo for the above application. I have been populating the document with the properties that are being extracted currently by shuttle and the ones that can be extracted using MediaMetaDataRetriever library. I'll add the metadata that is being extracted in the app to the document and make a rough comparison chart in the end. Here's the link to the document.

There's no sufficient information about how to build the project in the Readme files. I tried to compile the build but the AS couldn't find the proper configuration for it. I noticed that there are multiple modules that the AS recognizes as "runnable" but I think that's one of the reasons why the build fails.

barbeau commented 4 years ago

@BumbleFlash Thanks for putting this together! Given that it doesn't look like we gain much by potentially switching to libmediainfo, let's stick with the current Shuttle method (content provider) for reading metadata. We also know that the Shuttle method is reliable on Android as it's been widely used in production.

BumbleFlash commented 4 years ago

About saving events or collecting data, prior to our discussion in Teams, should we use a BroadcastReceiver similar to the implementation back in OBA? https://github.com/OneBusAway/onebusaway-android/blob/master/onebusaway-android/src/main/java/org/onebusaway/android/travelbehavior/receiver/TransitionBroadcastReceiver.java#L120

barbeau commented 4 years ago

No, we shouldn't need that in MUSER (that I know of at this point). BroadcastReceiver is required to get callbacks from Android for ActivityTransitions, which is why we used it in OBA. For MUSER, we shouldn't have any long-running tasks, so you can directly call the equivalent of TransitionBroadcastReceiver.saveTravelBehavior() within MUSER when events are triggered.

barbeau commented 4 years ago

To follow-up my above comment - the only case where we might use a BroadcastReceiver is if the Android platform broadcasts events for playing, pausing, etc. that we would need to intercept for logging purposes. But a BroadcastReceiver would probably already be implemented in Shuttle for these types of events, I would imagine.

BumbleFlash commented 4 years ago

IIRC Shuttle uses a more "function decomposition" method to manage to play, pause, and other events. Here's how it goes when the play or pause button is clicked. 1) The togglePlayback() is called from the PlayPauseView's OnClickListener. https://github.com/CUTR-at-USF/MUSER/blob/usf/app/src/main/java/com/simplecity/amp_library/ui/screens/nowplaying/PlayerFragment.java#L229 2) In the PlayerPresenter.kt, it further calls the togglePlayback() function from MediaManager class. https://github.com/CUTR-at-USF/MUSER/blob/usf/app/src/main/java/com/simplecity/amp_library/ui/screens/nowplaying/PlayerPresenter.kt#L178 3) The togglePlayback() from the MusicService is called in the MediaManagerclass. https://github.com/CUTR-at-USF/MUSER/blob/usf/app/src/main/java/com/simplecity/amp_library/playback/MediaManager.java#L188 4) Lastly, the togglePlayback() from PlaybackManager.java is called which handles the play or pause of the song. https://github.com/CUTR-at-USF/MUSER/blob/usf/app/src/main/java/com/simplecity/amp_library/playback/PlaybackManager.java#L537

barbeau commented 4 years ago

@BumbleFlash One option for us is to split UI actions and player events entirely.

So when the user starts and then pauses the player instead of having this stream of events:

...you'd have this stream of events:

And when the song plays to the end and then starts the next track without any user intervention, you'd have this stream of events without any UI actions:

A benefit of this is that we could track all player events (I think?) directly at the PlaybackManager class level regardless of whether a user interface action was involved, which prevents us from having to prevent PLAY events if PLAY_MANUAL triggered it. The main downside I see is that the stream is slightly more complex and would require more logic to process into higher level information.

An alternate approach with simpler post-processing would be to keep our current Event and EventType structure, but add another field to Event boolean userAction, with true if it was triggered by the user and false if it was triggered by the platform. So we'd get rid of PLAY_MANUAL and PAUSE_MANUAL, and instead have Event.userAction = true with EventType = PLAY, and Event.userAction = true with EventType = PAUSE.

So the first example stream would end up looking like:

Thoughts on this? I think the data model in the first example that completely separately EVENTs and UI_ACTIONs is cleaner than adding a boolean variable to EVENT and tracking everything there. I think as long as we have the nanoTime field for all UI_ACTIONs and EVENTs it shouldn't be too hard to combine them into a single ordered data structure in post processing.

BumbleFlash commented 4 years ago

The first example looks tidier to me too, however, actions like shuffle and repeat are implemented in the MusicService class level while the events that directly affect a song as you said, can be found at the PlaybackManager class level. That being said, we may need to monitor the flow in the MusicServiceif we're capturing the stream of events like

or

So, I am assuming, one way to go about storing UI_ACTIONs is by adding another enum to the pojo?

Also, should I open a separate issue on this so that we could have our discussion there? As you said, the implementation may look complex and can potentially have a lot to talk about.

barbeau commented 4 years ago

I don't think it's an issue even if some system events are split across the MusicService and PlaybackManager classes. The important part is not having to keep track of whether or not a UI action triggered a system event within the MUSER code base.

Let's keep the discussion here as it's central to the logging to Firebase question. I think the easiest approach is to treat them as separate enums and log them separately to Firebase.

At a high level, here's how I think about the split between player events and UI events:

Thinking more, I think treating both as type of "events" is better than calling one an event and one an action.

So for Firebase we would have two collections nested under each user ID, one for UI events and one for player events, with the following paths:

users/<uid>/player-events/<recordID>/playerEventData
users/<uid>/ui-events/<recordID>/uiEventData

For code implementation, in theory you could have an abstract class Event with subclasses PlayerEvent and UiEvent, but apparently you can't do inheritance with Kotlin data classes: https://stackoverflow.com/questions/26444145/extend-data-class-in-kotlin

So let's define a Event interface that includes the following fields that are common to all events:

 val currentTimeMs: Long,
 val nanoTime: Long,
 val startTime: Long,
 val elapsedTime: Long,
 val song: SongData

...and then EventType would become:

public enum PlayerEventType {
        PLAY,
        PAUSE,
        SKIP,
        REPEAT,
        SEEK
}

...with Event renamed to PlayerEvent that implements Event and adds a single field playerEventType.

And, we'd have a new enum UiEventType:

public enum UiEventType{
        PLAY,
        PAUSE,
        SKIP,
        REPEAT,
        SEEK,
        CREATE_PLAYLIST,
        SELECT_CATEGORY,
        SELECT_ARTIST
}

...and a new UiEvent class that implements Event and adds a single field uiEventType:

(val uiEventType: UiEventType,
 val currentTimeMs: Long,
 val nanoTime: Long,
 val startTime: Long,
 val elapsedTime: Long,
 val song: SongData
)

That would allow us to have a single list of events that all implement Event interface and contains both UiEvents and PlayerEvents, like:

List<Event> events = new ArrayList()<>;
// Load events from Firebase
...
// Loop through events
for (Event event : events) {
    if (event instanceOf PlayerEvent) {
        // Do something with player events
    }
    if (event instanceOf UiEvent) {
        // Do something with UI events
    }
    ...
}

One reason for splitting these so much is that I realized we probably want to track some UI events that don't have any effect on the player itself (e.g., creating a playlist, selecting a music category, selecting a playlist, selecting an artist), so the UiEventType will probably have a lot more enum values than the PlayerEventType. It seems strange to me to keep a single enum to track both types of events when most enums won't apply to one of the two types of events.

What do you think?

BumbleFlash commented 4 years ago

Thank you for taking the time to type this out. From what I understand from the initial reading, is that it sounds totally fine to keep things as separate as possible. However, I do have a few clarifications.

If what I'm understanding is right, the PlayerEvent class would some what look like:

 data class PlayerEvent
 (val playerEventType: PlayerEventType,
  override val currentTimeMs: Long,
  ....
  override val songData: SongData
  ) : Event

Is this the right way to implement the interfaces in data classes in Kotlin?

barbeau commented 4 years ago

Oh, I forgot to add that we should use Kotlin for the Event interface. There is some strange behavior where you can't directly implement Java interface methods using Kotlin data class fields: https://youtrack.jetbrains.com/issue/KT-6653?_ga=2.30406975.1494223917.1585591891-1137021041.1573759593

After that, yes, that looks right, although I'll have to admit I haven't properly done Kotlin interfaces with Kotlin data classes yet.

BumbleFlash commented 4 years ago

Sounds good. Thank you for the link. I guess we'll have to compile and test it out.

BumbleFlash commented 3 years ago

@barbeau While writing the code to collect the "SKIP" event, there's a situation where the song actually repeats when the "SKIP" button is collected while there is no other song to play in the queue. Should I keep the UI_EVENT to be "Skip" and the player event to be "repeat" or should I keep both of them as "SKIP"? Thanks

barbeau commented 3 years ago

I would have two player events:

...and one UI event:

I would always log a player PLAY event when the song starts playing, no matter what player event preceded it. That way if we want to look at metrics like "how many songs were started from the beginning at seek position 0" it will be easy to answer.

BumbleFlash commented 3 years ago

@barbeau Sounds good. I'll have this ready as soon as possible. Also, can I have separate PRs for separate events so that easy for you to review and test changes?

barbeau commented 3 years ago

Yes, separate PRs sounds good, thanks! If at least one PR is almost implemented, let's merge that prior to the next release w/ the target SDK 29.