Unable to use avg within controller when using filterrific

Seems related to #160. I want to be able to filter and sort by an average rating. That part seems to work, but it ends up borking when it comes to pagination. I initially thought this was a kaminari or will_paginate issue, but it looks like it's breaking on filterrific. Here's an example:

> @filterrific.find.page(params[:page])
  Course Load (51.9ms)  SELECT  courses.*,
            avg(reviews.rating) as avg_rating,
            avg(reviews.difficulty) as avg_difficulty,
            avg(reviews.work_required) as avg_work FROM "courses" LEFT OUTER JOIN "reviews" ON "reviews"."course_id" = "courses"."id" LEFT OUTER JOIN "schools" ON "schools"."id" = "courses"."school_id" GROUP BY courses.id, schools.id ORDER BY LOWER(courses.department) desc LIMIT $1 OFFSET $2
=> [#<Course:0x00007fa521406280
  id: 1,
  name: "Unicorns 101",
  number: "1001",
  department: "UNI",
  school_id: 1,
  created_at: Sun, 02 Jun 2019 20:17:12 UTC +00:00,
  updated_at: Wed, 05 Jun 2019 19:06:15 UTC +00:00>,
...
]

That looks like it is returning what I want. However, this is not returning what I want and resulting in incorrect sizes (11 vs 186):

> @filterrific.find.page(params[:page]).count
   (2.4ms)  SELECT COUNT(*) AS count_all, courses.id, schools.id AS courses_id_schools_id FROM "courses" LEFT OUTER JOIN "reviews" ON "reviews"."course_id" = "courses"."id" LEFT OUTER JOIN "schools" ON "schools"."id" = "courses"."school_id" GROUP BY courses.id, schools.id
  ↳ (pry):4
=> {9=>8,
 5=>9,
 11=>9,
 4=>10,
 3=>8,
 10=>9,
 8=>5,
 7=>8,
 6=>9,
 2=>8,
 1=>1}

vs if I use just @courses which still has the aggregate functions:

> @courses.page(params[:page]).size
  CACHE  (0.0ms)  SELECT COUNT(*) FROM "courses"
  ↳ (pry):6
=> 30

I recognize that this might be a cannot fix, but I'm unsure if I'm just doing something incorrectly. Here's my index method in the controller:

(@filterrific = initialize_filterrific(
        Course.all.with_averages,
        params[:filterrific],
        select_options: {
          sorted_by: Course.options_for_sorted_by,
          with_school_id: School.options_for_select
        }
      )) || return
@courses = @filterrific.find.page(params[:page])

jhund / filterrific

Unable to use avg within controller when using filterrific #182