Delivery Type: Extend domain to include credential

[jeff-grann]

Some QA approvals are specific to a credential's delivery type with different approvals required for online vs in-person credentials. To support those business processes, please consider extending the deliveryType domain to also include credentials.

@jkitchensSIUC Generally, a person has to complete a learning opp and/or assessment to earn a credential and the related learning opps and/or assessments are what’s delivered. The credential isn’t delivered until it’s earned and that’s described in the verification profile.

[siuc-nate]

I would reiterate the second paragraph there - the classes/tests required for a credential are what get delivered online or in-person (or both); I'm not sure the notion of delivery really applies to the credential itself (separate from those classes/tests).

A badge claim is usually online, but that is, as you mention, handled via verification profile as part of the organization.

[jeff-grann]

While it is true that delivery type applies to particular assessments and learning opportunities, it seems like delivery type should also be an option for the credential itself. For example, how could the delivery type for the credential be accurately inferred unless the provider has published all of their learning opportunities and assessments? Even with full publication, wouldn't users need to inspect each learning opportunity and assessment and then make an inference about the credential delivery type.

If extending delivery type to credentials is problematic, would a new property be needed?

[siuc-nate]

What I'm getting at is how the notion of delivery applies to an intangible thing like a credential - what does it mean that a credential is "delivered" online vs in person?

What would an example be of the credential itself being delivered rather than something that was done to earn the credential (assessment/learning opportunity) or to verify that someone has a credential (badge claim/verification service profile)?

[siuc-nate]

If what you're trying to get at is summary information (i.e. "the assessments and/or learning opportunities for this credential are all/generally/usually/etc delivered online") then maybe that makes a little more sense, but it also violates the idea that data should be published within the entity it really belongs to.

That said, we have violated that a few times now (the availability properties, cost properties, etc.) so it wouldn't be unheard of - though it will have the same effect that adding those properties had.

Taking "available online at" as an example: If you can say "the credential is available at [website]" and "the learning opportunity is available at [website]" then:

• Where do you, as the publisher, put the website? One, the other, or both?
  • If you update the information in one of your assessments/learning opportunities, you have to update all of the credentials that reference that assessment/learning opportunity
• Where do you as the person searching (i.e. the developer writing code) look for that information? One, the other, both?
  • How do you resolve mismatches (the website is different on one entity from another: which one is correct)?
  • What about missing data (the website exists on one entity but not the other: do you assume it should be there)?
  • What about duplicate data (if the website is on both entities, does that give that data more importance)?

Summarizing data like that should be the job of the intermediate system rather than the data itself, in my opinion, unless there is something about the data that explicitly defines it as being a summary - sort of how alignment object works: it basically says "this isn't the real home of this information, but if you were to look it up here's what you'd probably find".

But that discussion is probably a bit beyond the scope of this issue.

Anyway, since we've done it before, I guess I can't really justify arguing against doing it again, aside from my usual pushback against schema bloat. @stuartasutton what do you think?

[siuc-nate]

@stuartasutton @jeff-grann @jkitchensSIUC Jeanne and I are discussing this - let's either close it out or move forward with it by the August milestone deadline.

[stuartasutton]

I lean toward moving it forward to August.

[jeff-grann]

After discussion it's clear that CTDL supports making inferences about a credential's deliveryType via its associated Learning Opportunities and Assessments. Apps using registry information will need to consider their users' needs and develop supportive summery reports, such as the consistency of deliveryType across a credential.

[siuc-nate]

Per our 12/3/2018 meeting:

Proposal: Add ceterms:deliveryType to Credential

Delivery Type is a concept scheme. The concepts include:

• Blended Delivery
• Online Only
• In Person

Reasoning: Agencies would like to avoid publishing the learning opportunity profiles and/or assessment profiles for credentials and would rather just indicate the delivery type for such learning opportunities/assessments as part of the data for the credential itself. This came about as a result of an agency wanting to indicate that their credentials were offered online.

Counter: Delivery Type is intended to describe the learning opportunity/assessment, not the credential. Instead publish the credential and its related learning opportunity/assessment (including the delivery type) and connect them via ceterms:requires. For shorthand publishing, use ceterms:availableOnlineAt.

[stuartasutton]

Can we hold this conversation from going further for a few days--until Jeanne and I (and others) have gotten past Learn & Build and I can comment sufficiently. I've just settled into the hotel here in DC and need to get to bed. This has to wait until Thursday.

[jeff-grann]

The L&B discussion resulted in the following approach, which seemed acceptable at the time as way to balance the needs for a consistent data model, CE minimum data policy, and our partners' capacity.

Credential providers wishing to differentiate their programs by delivery type may use the existing learning opportunity profile (i.e., no data model change) for their program. For those providers only wishing to denote the delivery type of their program, Credential Engine will provide default entries for all other required fields (i.e., no change to minimum data policy). As partners utilize this approach, Credential Engine will review the user experience and work to optimize the publishing workflow.

[siuc-nate]

Per our 12/6/2018 call: The following solutions are intended to communicate the delivery types for the learning opportunities and/or assessments required to earn a credential, without including ceterms:deliveryType as a property of Credential.

New Solutions 1 and 2 would benefit searching by surfacing relevant information.

Current/Existing term/property: Entering a URL for ceterms:availableOnlineAt, indicates that the credential is available online. The filters for this are: Publisher/Finder:

• Use the "Is Available Online" filter (under Reports) Search API:
• Use { "ceterms:availableOnlineAt": "search:anyValue" }

The current definition of ceterms:availableOnlineAt does not clearly indicate that the resource is delivered online and needs to be revised. We currently have a data quality issue where users are entering URLs that are more appropriate for the ceterms:subjectWebpage property and where the credential's assessment or learning opportunity are not necessarily delivered online. There is also a problem where the real URL is not publicly available, so an incorrect URL (that often repeats general information or registration information) is being used instead. The requirement to enter a URL is a limitation when no real URL is available.

Current definition: "Online location where the credential, assessment, or learning opportunity can be pursued." Proposed definition: "The URL to start pursuing a credential where its assessment and/or learning opportunity are delivered online."

In conclusion, users are using this URL because they have no choice when they want to indicate that the delivery type for requirements for the credential are delivered online. Additionally, there is no clear indication that the credential and its requirements are actually delivered online (as opposed to, for example, a signup form for an in-person course being online).

Expecting users to enter a URL rather than a property that has clear intent, will provide inaccurate and inconsistet information.

New Proposed solution 1: Create a concept scheme to communicate this information, via a property on Credential: Property: ceterms:requirementsDeliveryType Definition: The delivery types for the activities required to earn the credential. Concept Scheme: ceterms:RequirementsDelivery Concepts:

• requirementsDelivery:HasOnlineLearningOpportunity
• requirementsDelivery:HasInPersonLearningOpportunity
• requirementsDelivery:HasBlendedLearningOpportunity
• requirementsDelivery:HasOnlineAssessment
• requirementsDelivery:HasInPersonAssessment
• requirementsDelivery:HasBlendedAssessment

New Proposed solution 2: Add a set of boolean properties to Credential:

• ceterms:hasOnlineLearningOpportunity
• ceterms:hasInPersonLearningOpportunity
• ceterms:hasBlendedLearningOpportunity
• ceterms:hasOnlineAssessment
• ceterms:hasInPersonAssessment
• ceterms:hasBlendedAssessment

Facilitating easier entry of learning opportunities (and possibly assessments): Having properties that enable describing what is needed to earn a credential is more appropriate than forcing users to create an assessment and/or learning opportunity where most if not all of the information would be duplicative. However, the process described here could be enabled independently of solutions 1 and 2. This process can be considered as a way to clone information to save time typing rather than a solution just to solve the deliveryType problem. This issue came about because creating a learning opportunity just to provide the delivery type would entail (due to the minimum data policy) also adding a lot of information that would mostly if not entirely be duplicative with information provided on the Credential. This process aims to make that easier to do. Ideally the relevant data would instead be on Credential instead of going to the lengths described below for the sake of a single property. Enable (via the publisher (editor and bulk upload)) the user to easily clone/duplicate relevant information from the credential to an assessment and/or learning opportunity that would then be automatically added as a required learning opportunity when the credential is created in the publisher.

• Name
• Description
• Available At
• Available Online At
• Availability Listing
• Delivery Type
• CIP Code

Pros:

• The assessment and/or learning opportunity would exist, already linked to the credential
• The assessment and/or learning opportunity could then be edited later with additional information (such as competencies)

Cons:

• The information would be duplicated in two top-level documents in the registry

Requirements:

• System:
  • The system must avoid creating an additional generated assessment and/or learning opportunity rather than updating the initially-generated one, even if the user made other manual updates to it.

Workflow (new credential):

• The user creates a credential
• The user selects whether or not they want to duplicate the data for the properties listed above to a newly-generated assessment and/or learning opportunity

Workflow (existing credential):

• The user edits a credential
• The system will prevent automatically updating any existing assessment and/or learning opportunity
• The user selects whether or not they want to duplicate the data for the properties listed above to a newly-generated assessment and/or learning opportunity

[Lomilar]

Echoing some sentiment from the conversation at the Learn and Build summit:

The user selects whether or not they want to duplicate the data for the properties listed above to a newly-generated assessment and/or learning opportunity

Pretty much all database normal forms include: "Don't duplicate data if you can at all help it."

and that denormalization (say, duplicating data) should be carried out for scalability or performance reasons, not for ease of use or ease of understanding. Computers and developers are good at managing complexity.

If a credential has an online learning opportunity (per, say, the boolean property) but doesn't have an online learning opportunity (described and linked) then is it out of compliance? Does this need validation? If I check that there is an online, blended, and live learning opportunity and then only provide the live one, which is shown in the finder?

These are the complexities that arise from duplicating data, is that there is also a duplication of claims of truth that can be immediately conflicting. Every program that consumes the data must wrestle with these questions independently, in that they must now either believe the Credential data or believe the attached Learning Opportunity data.

It is much easier to just traverse over to the learning opportunity and see if it is offered online, and even if the learning opportunity is ill defined or missing data, it would still be better than allowing conflicting information.

[siuc-nate]

@Lomilar in the case described above, there would be no published learning opportunity (or assessment). Publishing this data is seen by some of our partners as too high of a barrier to entry, so we are trying to determine a way to make it easy for them to publish credentials without it. I personally do not like it either, but I do admit that a lot of the information we get when such a learning opportunity is published is often very minimal and/or duplicitous with what's in the credential anyway (most likely due to many conceptualizations of the data not separating the credential from its program, as we often see). So there is a practical problem in addition to the complexity of publishing multiple documents.

That said, you raise a good point with regards to data quality - when the credential and the learning opportunity both exist but do not "agree", which one should be considered authoritative (particularly if the credential and the learning opportunity have different owners)? I suppose it's easy enough to assert that the credential should be authoritative about anything describing the credential, but then does that imply a need to provide a way to indicate (to a system) which properties are about something else, and thus okay for a consuming system to "disbelieve" in the face of a conflict?

Honestly, after working through all of those proposals outlined above, I think just adding deliveryType to Credential is a far, far simpler solution - yes, it violates the notion of putting the data where it truly belongs, but we have done this already with a number of other properties (and are currently in the process of doing so with regards to adding instructionalProgramType to Credential), and (as with instructionalProgramType) it accommodates the very real problem of partners not wanting to publish additional standalone profiles (and of those profiles being mostly duplicate data when they are published).

[Lomilar]

The minimum data requirements prevent considering a third option, which is not requiring name and description on learning opportunities, so it can be articulated: "There exists a learning opportunity for this credential that is online." and nothing else. All learning opportunity fields that exist on a credential during the import (and if necessary in the ui/ux) process could create this very shallow learning opportunity, and then removing the duplicate data would have a path forward as well.

Going down the argument rabbit hole a bit:

• If people are duplicating data, they are probably doing so because data that kinda looks like 'that' is required in two places. This is why some linked data schemata don't require things, because consuming applications (like the finder) are the ones that have requirements (like the need to display at the credential level whether it has an online program) and that must filter out data that doesn't meet their requirements, not the linked data itself. Placing requirements on linked data itself creates a sort of supply side bottleneck that encourages bad (duplicate, placeholder) data instead of missing data. Not entirely sure which one is worse, but I prefer missing data over bad data.

• The place for data to percolate from children up to the parents (for performance, display, or whatever reason) is in the search index of the Finder or the consuming application. The learning opportunity (when displayed in the finder) could easily fetch its name and description from the credential if it doesn't exist in the opportunity, and mark it as "Learning opportunity for: " or some other demand side compromise.

• Saying that "Oh, this data appears in two places but you should trust it more when it is here than over there" is a red flag. See CASE for an example. Why is the name of everything duplicated in every link? If I split this data up, do I trust the name in the link or the name of the thing at the destination URI? Why is the CFPackage Title the same as the CFDocument Title? What is the difference between those?

[stuartasutton]

All I am doing here in this issue is addressing the minimum data policy aspect at the root of problem. Other than that, I certainly prefer the "Proposal solution 1" (concept scheme) to the second boolean solution.

@siuc-nate says above (and @jkitchensSIUC said frequently in D.C.):

"This issue came about because creating a learning opportunity just to provide the delivery type would entail (due to the minimum data policy) also adding a lot of information that would mostly if not entirely be duplicative with information provided on the Credential. "

On the long flight and shuttle home from D.C., I thought about this. My conclusion is that there is no "minimum data policy" problem or violation. If we distinguish between describing something and simply representing it in some loose form, then we would not have a minimum data problem.

We just went through a tedious process of making it possible to loosely identify an entity via bnode when we have no description (record) of that entity in the registry. Where we do so, those bnodes do not follow the minimum data requirements for the entity the bnode represents because we are not trying to fully describe the entity. We have the exact same problem here. We need to make an assertion about something that does not exist in the registry; so, we can use bnode with minimal data.

Going back to discussions around the recent update to the data representation in the registry, we see examples such as this in closed issue #394:

@prefix ceterms: <http://purl.org/ctdl/terms/> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xml: <http://www.w3.org/XML/1998/namespace> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

<http://credentialEngineRegistry.org/resources/CE-D680FE7F-CF61-41DC-841E-3A2C91BF96BC> a ceterms:BachelorDegree ;
    ceterms:ctid "CE-D680FE7F-CF61-41DC-841E-3A2C91BF96BC" ;
    ceterms:name "Brewing Certificate"@en-US ;
    ceterms:accreditedBy [ a ceterms:QACredentialOrganization ;
        ceterms:name "Northwest Region Accreditor of Breweries"@en-US ;
        ceterms:subjectWebpage <http://NorthWestBrewers/accreditation> ]

The ceterms:name and ceterms:subjectWebpage in the ceterms:QACredentialOrganization don't come close to satisfying the minimum data requirements for ceterms:BachelorDegree; but it is OK because we are NOT describing the degree but simply referencing it through one or more properties where we do have available data.

So, in like fashion, the following would be perfectly OK even though lacking almost all of the properties in the minimum data requirements for ceterms:LearningOpportunityProfile:

@prefix ceterms: <http://purl.org/ctdl/terms/> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xml: <http://www.w3.org/XML/1998/namespace> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

<http://credentialEngineRegistry.org/resources/CE-D680FE7F-CF61-41DC-841E-3A2C91BF96BC> a ceterms:BachelorDegree ;
    ceterms:ctid "CE-D680FE7F-CF61-41DC-841E-3A2C91BF96BC" ;
    ceterms:name "Brewing Certificate"@en-US ;
    ceterms:requires [ a ceterms:LearningOpportunityProfile ;
        ceterms:availableOnlineAt  <http://NorthWestBrewers/LearnOpp> ]

Since it's a bnode,

1. it doesn't show up in searches etc. but could support rolling up the online availability data by downstream apps (e.g., the Finder) to display it at the credential level or wherever; and
2. we don't need to insert any duplicate (and confusing) boilerplate or placeholder data to satisfy an inapplicable minimum data requirement.

[Lomilar]

I am happy with @stuartasutton's suggestion above. It generates some coding hoops (having to check whether a ceterms:requires is a bnode or a URL that needs fetching) but I don't think we can get away from some sort of hoop to jump through in this space.

[siuc-nate]

@stuartasutton @jkitchensSIUC Once again, bnode saves the day (in a rather obvious-in-hindsight, "why didn't we think of that sooner" sort of way).

I think this solution could extend to other problems, potentially solving other cases where we've solved similar problems by adding properties to credential that don't belong on credential (and may lead to an exploration of properties that could be removed from credential because they really do belong somewhere else).

In practice though, it would present a couple of interesting problems (on top of the usual development workload):

• Partners would still need be willing to publish the bnodes (though they could do this in the same transaction as the credential)
• It may complicate the editor and/or bulk upload tools, as those will need to basically offer everything that is in the "full" version of the assessment/learning opportunity tools, but presented in a context where it's clear that such things are being published as bnodes rather than full, standalone entities

That said, if we did go with the bnode-based solution, we would not need to create either the concept scheme or the boolean properties, correct?

[siuc-nate]

Two new properties on Credential that indicate the delivery type of the assessment or learning opportunity to earn the credential will be added for the December 2018 release.

• Reuse the existing Delivery vocabulary as values for both of these new delivery type properties.

These properties will be added to the minimum data policy as "Recommended."

Actions to be taken: Add:

URI: ceterms:assessmentDeliveryType Label: Assessment Delivery Type Definition: Delivery type for the assessment for the credential. Comment: Indicates the delivery type for the assessment for the credential. Domain: ceterms:Credential (and subclasses) Range: ceterms:CredentialAlignmentObject Target Vocabulary: ceterms:Delivery Subproperty Of: ceterms:deliveryType

Add:

URI: ceterms:learningDeliveryType Label: Learning Delivery Type Definition: Delivery type for the learning opportunity for the credential. Comment: Indicates the delivery type for the learning opportunity for the credential. Domain: ceterms:Credential (and subclasses) Range: ceterms:CredentialAlignmentObject Target Vocabulary: ceterms:Delivery Subproperty Of: ceterms:deliveryType

[siuc-nate]

These terms have been added to pending CTDL and noted in the history tracking.