Run beta test on Survey with 1-2 Providers

[jmakowski1123]

Take the FormAssembly Survey and the Google sheets Survey for a test drive with 1-2 Providers and gather feedback/possible changes and edits.

[jmakowski1123]

2/23 - Gabriel from OpenCraft and Carolina+Juan from eduNEXT are doing test runs. Confirmed during Marketing WG and follow up email with instructions sent.

[jmakowski1123]

Feedback from Juan:

Re: The FA survey for single instances:

The queries were very clear and easy to run for the single tenant case.

The way the query computes active courses includes any course that started and has not yet finished, regardless of the number of learners enrolled. in our test, this was a bit misleading, because the number of courses that fit that criteria, but had virtually no enrollments was huge. If we changed the script to this one: select count() from course_overviews_courseoverview where START < now() and (end > now() OR end is NULL) AND (SELECT COUNT() FROM student_courseenrollment WHERE student_courseenrollment.course_id=course_overviews_courseoverview.id)>5

the number or courses will drop substantially.

I like the way active users are being computed (the ones that logged in recently), but I feel the opportunity to learn how many learners have been reached overall is being missed, due to the lack of an additional query for total_learners: ( select count(*) from auth_user where is_active=1 ) as total_users

I believe addin this bit will be usefull and even more so in combination with the total number of enrollments (which is already being queried). I feel this would be a great opportunity to ask which of the components of the platform are being used other than the LMS and studio. It could be at the application level (ecommerce, insights, mobile apps) or perhaps even at a more fine grained level (course discovery service, persistent grades, gradebook, microfrontends, etc)

[jmakowski1123]

Feedback from Juan:

Re: mult-tenancy excel:

Now, for the multi tenant experience things got a bit harder. I guess you can differentiate 2 types of multi tenant instances:

1. The ones run by a single large entity where having multiple tenants makes sense because they run multiple different initiatives (for example a university with a tenant for each department).
2. the ones run by a provider of open edX hosting in a SaaS fashion where each tennant is efectively a different organization.

For the first ones, it may not be difficult to fill in the survey a few times, and the interface is quite easy to handle, but the queries don´t split the data per tenant, so they will have to manually figure out or estimate the 4 numerical values without the provided queries .

For the second ones, (and the number here is probably very small), it's quite impractical due to the number of tenants at play. we have several thousands and I know appsembler has at least a few thousands as well.
The spreadsheet approach didn´t help that much, because most of the qualitative fields are not in the database so a lot of manyal work would be involved and the quantitative ones (the ones the queries are for) face the same issue I mentioned before of not splitting the numbers by tenant.

After pondering for a while I'd recommend: a) To include the first type of multi tenants with the normal instances, and just add a couple of items to the query that would help understand the extent of the multi tenancy being used, namely: -total number of orgs in organizations_organization -total number of sites in site_configuration_siteconfiguration

b) To treat the SaaS provider's multi tenants as aggregators in its own right and have them run the querys and fill the survey only once. perhaps this will require to add an option to the "this site serves" question such as "Multiple sectors simultaneously".

c) to add a question that would help you deduplicate the information of someone that is being hosted by a saas provider, and simply gets to the survey through the community channels. this question will potentially be valuable for tCRIL as well: What hosting arrangment is your site using? self hosted by my organization | hosted by a saas provider | hosted by a service provider on premise | other The redaction will surely need improvent, but you get the idea :)

[jmakowski1123]

Juan - Two additional pieces of feedback:

Re: # of unique courses: I'd suggest to break this in 2 datapoints, number of total courses and number of courses delivered last year (2021).

Re: Total number of registered user: break into total number of registered learners, and number of learner using the site in 2021

[feanil]

@jmakowski1123 some initial thoughts:

• Recommendations based on Feedback
  • For the existing course count query, update it to only count courses with more than 5 enrollments.
  • Add a new question for total historical learner count.
  • Add a query for the number of orgs and number of sites in site_configuration.
• Other thoughts
  • Not sure what to do about the places running a thousand instances.

Do you have feedback from Open-Craft yet?

[jmakowski1123]

@jmakowski1123 some initial thoughts:

• Recommendations based on Feedback

  • For the existing course count query, update it to only count courses with more than 5 enrollments.
  • Add a new question for total historical learner count.
  • Add a query for the number of orgs and number of sites in site_configuration.

• Other thoughts

  • Not sure what to do about the places running a thousand instances.

Do you have feedback from Open-Craft yet?

@feanil Nothing yet from OpenCraft. I know Karim is working on it. If we don't have feedback by Thurs morning, I think we should go ahead with the first round of suggestions from Juan and can always circle back.

Your summary of the first three bullets sounds right to me. Re: sites with thousands, I'm almost inclined to say we try Option B and work 1:1 with those Providers (I think there are <10) to answer these questions manually:

b) To treat the SaaS provider's multi tenants as aggregators in its own right and have them run the querys and fill the survey only once. perhaps this will require to add an option to the "this site serves" question such as "Multiple sectors simultaneously".

[feanil]

@jmakowski1123 I updated the instructions, and the spreadsheet but don't have permission to update the form assembly form so you'll have to take car of that.

Updated Instructions Updated Google Sheet Template

[jmakowski1123]

Updated FormAssembly and tweaked google sheet so everything aligns. This should be ready to send now.