Fix Hyrax Analytics Reporting Layer

[labradford]

Story

Hyrax Analytics reporting layer is broken since GA4 updates. We need to fix this layer or work with the community to see when this will be done

Acceptance Criteria

• [ ] Hyrax reporting layer for GA4 is working

Screenshots / Video

Testing Instructions and Sample Files

-

Notes

Check with community before working on this ticket

[alishaevn]

may or may not be related: as of june 23. daniel and ben were looking into going in another direction other than legato since legato isn't being maintained anymore.

ref: thread.

[jeremyf]

The Hyrax::Analytics::Google defines the interface for Google Analytics. The public methods then use classes part of the Hyrax::Analytics::Google module to generate results. Those classes extend the Legato rubygem; a gem that the owner has said he doesn’t have time to maintain.

One option is to help in that maintenance. Another option is to leverage https://github.com/googleapis/google-cloud-ruby, which includes https://rubygems.org/gems/google-analytics-data/versions/0.2.0.

Below are the locations where we use methods from Hyrax::Analytics. The goal is to gain insight into the expected data structure of the returned elements; the expected data structure is not immediately obvious.

spec/features/work_show_spec.rb:6:    allow(Hyrax::Analytics.config).to receive(:analytics_id).and_return('UA-XXXXXXXX')
app/views/shared/_matomo.html.erb:7:    var u="<%= Hyrax::Analytics.config.base_url %>";
app/views/shared/_matomo.html.erb:9:    _paq.push(['setSiteId', "<%= Hyrax::Analytics.config.site_id %>"]);
app/views/shared/_ga.html.erb:3:  _gaq.push(['_setAccount', '<%= Hyrax::Analytics.config.analytics_id %>']);
app/views/hyrax/dashboard/_user_activity.html.erb:17:            <p><%= t('.new_visitors') %>: <%= Hyrax::Analytics.new_visitors('range', "#{@start_date},#{@end_date}") %></p>
app/views/hyrax/dashboard/_user_activity.html.erb:18:            <p><%= t('.returning_visitors') %>: <%= Hyrax::Analytics.returning_visitors('range', "#{@start_date},#{@end_date}") %></p>
app/views/hyrax/dashboard/_user_activity.html.erb:19:            <p><%= t('.total_visitors') %>: <%= Hyrax::Analytics.total_visitors('range', "#{@start_date},#{@end_date}") %></p>
app/views/hyrax/dashboard/_user_activity_graph.html.erb:3:  <% return_visits = Hyrax::Analytics.returning_visits_by_day("#{@start_date},#{@end_date}").list.reverse %>
app/views/hyrax/dashboard/_user_activity_graph.html.erb:4:  <% new_visits = Hyrax::Analytics.new_visits_by_day("#{@start_date},#{@end_date}").list.reverse %>
app/views/hyrax/admin/analytics/work_reports/_work_files.html.erb:24:            <td><%= Hyrax::Analytics.daily_events_for_id(@document.id, 'file_set_in_work_download').all %></td>
app/models/file_download_stat.rb:7:    # Hyrax::Download is sent to Hyrax::Analytics.profile as #hyrax__download
app/models/file_download_stat.rb:10:      profile = Hyrax::Analytics.profile
app/models/hyrax/statistic.rb:28:      # Hyrax::Download is sent to Hyrax::Analytics.profile as #hyrax__download
app/models/hyrax/statistic.rb:32:        profile = Hyrax::Analytics.profile
app/controllers/hyrax/stats_controller.rb:11:      @pageviews = Hyrax::Analytics.daily_events_for_id(@document.id, 'work-view')
app/controllers/hyrax/stats_controller.rb:12:      @downloads = Hyrax::Analytics.daily_events_for_id(@document.id, 'file-set-in-work-download')
app/controllers/hyrax/admin/analytics/collection_reports_controller.rb:10:          @pageviews = Hyrax::Analytics.daily_events('collection-page-view')
app/controllers/hyrax/admin/analytics/collection_reports_controller.rb:11:          @work_page_views = Hyrax::Analytics.daily_events('work-in-collection-view')
app/controllers/hyrax/admin/analytics/collection_reports_controller.rb:12:          @downloads = Hyrax::Analytics.daily_events('work-in-collection-download')
app/controllers/hyrax/admin/analytics/collection_reports_controller.rb:13:          @all_top_collections = Hyrax::Analytics.top_events('work-in-collection-view', date_range)
app/controllers/hyrax/admin/analytics/collection_reports_controller.rb:15:          @top_downloads = Hyrax::Analytics.top_events('work-in-collection-download', date_range)
app/controllers/hyrax/admin/analytics/collection_reports_controller.rb:16:          @top_collection_pages = Hyrax::Analytics.top_events('collection-page-view', date_range)
app/controllers/hyrax/admin/analytics/collection_reports_controller.rb:26:          @pageviews = Hyrax::Analytics.daily_events_for_id(@document.id, 'collection-page-view')
app/controllers/hyrax/admin/analytics/collection_reports_controller.rb:27:          @work_page_views = Hyrax::Analytics.daily_events_for_id(@document.id, 'work-in-collection-view')
app/controllers/hyrax/admin/analytics/collection_reports_controller.rb:28:          @uniques = Hyrax::Analytics.unique_visitors_for_id(@document.id)
app/controllers/hyrax/admin/analytics/collection_reports_controller.rb:29:          @downloads = Hyrax::Analytics.daily_events_for_id(@document.id, 'work-in-collection-download')
app/controllers/hyrax/admin/analytics/work_reports_controller.rb:18:            @pageviews = Hyrax::Analytics.daily_events('work-view')
app/controllers/hyrax/admin/analytics/work_reports_controller.rb:19:            @downloads = Hyrax::Analytics.daily_events('file-set-download')
app/controllers/hyrax/admin/analytics/work_reports_controller.rb:29:          @pageviews = Hyrax::Analytics.daily_events_for_id(@document.id, 'work-view')
app/controllers/hyrax/admin/analytics/work_reports_controller.rb:30:          @uniques = Hyrax::Analytics.unique_visitors_for_id(@document.id)
app/controllers/hyrax/admin/analytics/work_reports_controller.rb:31:          @downloads = Hyrax::Analytics.daily_events_for_id(@document.id, 'file_set_in_work_download')
app/controllers/hyrax/admin/analytics/work_reports_controller.rb:73:          @top_analytics_works ||= Hyrax::Analytics.top_events('work-view', date_range)
app/controllers/hyrax/admin/analytics/work_reports_controller.rb:77:          @top_analytics_downloads ||= Hyrax::Analytics.top_events('file-set-in-work-download', date_range)
app/controllers/hyrax/admin/analytics/work_reports_controller.rb:81:          @top_analytics_file_sets ||= Hyrax::Analytics.top_events('file-set-download', date_range)

We need to consider how the Hyrax::Statistic.ga_statistics is used. The one place is where we set ga_stats.

models/file_download_stat.rb:9:    def ga_statistics(start_date, file)
models/hyrax/statistic.rb:30:      def ga_statistics(start_date, object)
models/hyrax/statistic.rb:83:          ga_stats = ga_statistics(stat_cache_info[:ga_start_date], object)

The ga_stats is defined in Hyrax::Statistic#combined_stats which is used with Hyrax::Statistic

app/models/hyrax/statistic.rb:25:        combined_stats object, start_date, cache_column, event_type, user_id
app/models/hyrax/statistic.rb:79:      def combined_stats(object, start_date, object_method, ga_key, user_id = nil)

The combined_stats method call is in Hyrax::Analytics#statistics.

❯ rg "\.statistics" app
app/controllers/concerns/hyrax/admin/stats_behavior.rb
32:        add_breadcrumb t(:'hyrax.admin.sidebar.statistics'), hyrax.admin_stats_path

app/services/hyrax/user_stat_importer.rb
72:      rescue_and_retry("Retried #{from} on #{user} for #{object.class} #{object.id} too many times.") { from.statistics(object, start_date, user.id) }

app/views/hyrax/dashboard/_repository_growth.html.erb
8:  Hyrax.statistics.repositoryGrowth = <%== repository_growth %>

app/views/hyrax/dashboard/_visibility_graph.html.erb
10:        Hyrax.statistics.repositoryObjects = <%== repository_objects %>

app/views/hyrax/dashboard/sidebar/_activity.html.erb
32:      <span class="fa fa-bar-chart" aria-hidden="true"></span> <span class="sidebar-action-text"><%= t('hyrax.admin.sidebar.statistics') %></span>

app/assets/javascripts/hyrax/app.js.erb
66:        new AdminGraphs(Hyrax.statistics);

app/presenters/hyrax/work_usage.rb
31:      to_flots WorkViewStat.statistics(model, created, user_id)

app/presenters/hyrax/file_usage.rb
32:      to_flots(FileDownloadStat.statistics(model, created, user_id))
36:      to_flots(FileViewStat.statistics(model, created, user_id))

We can, for now, ignore lines that have Hyrax.statistics as those are javascript variables; but we’ll need to review what repository_growth returns as the structure might be relevant.

We can ignore the translation entries (e.g. t('hyrax.admin.sidebar.statistics') as those aren’t data structures.

This leaves the Hyrax::FileUsage, Hyrax::WorkUsage, and Hyrax::UserStatImporter to review.

[jeremyf]

Overview of what’s done:

• I have connected to PALS demo analytics and verified that I could fetch data from Google analytics.
• I have also written and tested the collation of that Google Analytics into the Hyrax::CounterMetric format, which feeds into Sushi.
• The new gem (Hyrax::Analytics::G4) is ready to work with different versions of Hyrax; namely those that implement a Hyrax::CounterMetric model.
• I have implemented prototypes of jobs (both for Hyku applications and non-Hyku Hyrax-based applications).
• I encountered problems regarding application dependencies (namely the Faraday gem); the Google provided library for fetching data needs a much newer version than the specified versions of several dependencies in PALS’s Hyku instance. More on this later.
• I have worked around the above problem by creating a Fake analytics generator (see Hyrax::Analytics::G4 README); which means that we can test how we map that data to Hyrax::CounterMetric.

What remains:

• I’m uncertain how we’ll store the credentials necessary for querying Google Analytics. It’s expected to be a Ruby hash, so we can construct that based on ENV vars as these are secret.
• I’m uncertain how we’ll expose the Google Analytics property associated with the tenant/roll-up analytics account. This is not the G-RANDO123 that we see in the site. This property is not a secret and may be valid for more than one tenant.
  • If a tenant does not share analytics with a roll-up account, we’ll need to consider how to provide per-tenant secrets (something that may have existing patterns).
• Integrate fetching live data, either by:
  • Updating the Faraday dependencies in the application OR
  • Replacing Google’s provided library with something else
• Fixing the Hyrax Analytics data layer; the SUSHI metrics requires different fetching logic than the Hyrax UI’s analytics uses. There is commonality between approaches, but the realities are different enough.