SBinary style for Reader/Writer

[ezhulenev]

sbinary (https://github.com/harrah/sbinary) uses much more cleaner style of writing custom Reads/Writes formats

see example at DataApi proejct: DatascopeProtocol.scala

[mandubian]

Which part exactly do you find too complicated?

[ezhulenev]

Here is my nested structure that I load from XML and want to push all this data as facts into Datomic for initial load, and I'm trying to write Reader/Writer, and it looks ugly ;)

case class Company(repNo: String,
                   name: Option[String],
                   xref: Xref,
                   status: Status,
                   currencies: Currencies,
                   industryClassifications: IndustryClassifications)

object Company {

  case class Xref(IRSNo: Option[String],
                  CIKNo: Option[String],
                  FProXRef: Option[String],
                  OrgID: Option[String])

  case class Status(active: Boolean,
                    currentEstimates: Boolean,
                    companyType: String,
                    availableInfo: AvailableInfo,
                    filingStatus: FilingStatus,
                    hasPublicDebt: Boolean)

  case class FilingStatus(currentStatus: String, reason: BigInt)

  case class AvailableInfo(BusinessIntelligence: Boolean,
                           Financials: Boolean,
                           Estimates: Boolean,
                           SignificantDevelopment: Boolean,
                           Officers: Boolean)

  case class Currencies(financialStatements: Option[String],
                        estimates: Option[String],
                        primaryIssuePrice: Option[String])

  case class IndustryClassifications(TRBC: Option[IndustryClassification],
                                     NAICS1997: Option[IndustryClassification],
                                     SIC1987: Option[IndustryClassification])

  case class IndustryClassification(code: String,
                                    mnemonic: Option[String],
                                    description: Option[String])
}

[ezhulenev]

I've got 26 possible attributes, which is greater then 22 limit for case class, and it breaks builder pattern for Reader/Writer

    val repNo                   = Attribute(ns.company / "repNo",                                        SchemaType.string,  Cardinality.one).withUnique(Unique.identity)
    val name                    = Attribute(ns.company / "name",                                         SchemaType.string,  Cardinality.one)
    val irsNo                   = Attribute(ns.company.xref / "IRSNo",                                   SchemaType.string,  Cardinality.one)
    val cikNo                   = Attribute(ns.company.xref / "CIKNo",                                   SchemaType.string,  Cardinality.one)
    val fProXRef                = Attribute(ns.company.xref / "FProXRef",                                SchemaType.string,  Cardinality.one)
    val orgId                   = Attribute(ns.company.xref / "OrgID",                                   SchemaType.string,  Cardinality.one)
    val active                  = Attribute(ns.company.status / "active",                                SchemaType.string,  Cardinality.one)
    val currentEstimates        = Attribute(ns.company.status / "currentEstimates",                      SchemaType.string,  Cardinality.one)
    val businessIntelligenceA   = Attribute(ns.company.status.availableInfo / "businessIntelligence",    SchemaType.boolean, Cardinality.one)
    val financialsA             = Attribute(ns.company.status.availableInfo / "financials",              SchemaType.boolean, Cardinality.one)
    val estimatesA              = Attribute(ns.company.status.availableInfo / "estimates",               SchemaType.boolean, Cardinality.one)
    val significantDevelopmentA = Attribute(ns.company.status.availableInfo / "significantDevelopment",  SchemaType.boolean, Cardinality.one)
    val officersA               = Attribute(ns.company.status.availableInfo / "officers",                SchemaType.boolean, Cardinality.one)
    val currentStatus           = Attribute(ns.company.status.filingStatus / "currentStatus",            SchemaType.string, Cardinality.one)
    val reason                  = Attribute(ns.company.status.filingStatus / "reason",                   SchemaType.bigint, Cardinality.one)
    val hasPublicDebt           = Attribute(ns.company.status / "hasPublicDebt",                         SchemaType.boolean, Cardinality.one)
    val financialStatementsC    = Attribute(ns.company.currencies / "financialStatements",               SchemaType.string,  Cardinality.one)
    val estimatesC              = Attribute(ns.company.currencies / "estimates",                         SchemaType.string,  Cardinality.one)
    val primaryIssuePriceC      = Attribute(ns.company.currencies / "primaryIssuePrice",                 SchemaType.string,  Cardinality.one)
    val trbcCode                = Attribute(ns.company.industryClassification.TRBC / "code",             SchemaType.string,  Cardinality.one)
    val trbcMnemonic            = Attribute(ns.company.industryClassification.TRBC / "mnemonic",         SchemaType.string,  Cardinality.one)
    val trbcDescription         = Attribute(ns.company.industryClassification.TRBC / "description",      SchemaType.string,  Cardinality.one)
    val naics1997Code           = Attribute(ns.company.industryClassification.NAICS1997 / "code",        SchemaType.string,  Cardinality.one)
    val naics1997Mnemonic       = Attribute(ns.company.industryClassification.NAICS1997 / "mnemonic",    SchemaType.string,  Cardinality.one)
    val naics1997Description    = Attribute(ns.company.industryClassification.NAICS1997 / "description", SchemaType.string,  Cardinality.one)
    val sic1987Code             = Attribute(ns.company.industryClassification.SIC1987 / "code",          SchemaType.string,  Cardinality.one)
    val sic1987Mnemonic         = Attribute(ns.company.industryClassification.SIC1987 / "mnemonic",      SchemaType.string,  Cardinality.one)
    val sic1987Description      = Attribute(ns.company.industryClassification.SIC1987 / "description",   SchemaType.string,  Cardinality.one)

[mandubian]

I think macros would fit well here... but they aren't written :( have you had a look at sample 3 in play-datomisca? Having all attributes (which are mandatory for schema), you can write Read/Write quite easily...

[ezhulenev]

Yes, I'm trying to build repository based on this example. Hopefully finish today with some basic read/write so you can take a look at a code tomorrow morning and say what you think about it ;)

[mandubian]

no pb ;)

[ezhulenev]

@pascal I've pushed some code into fundamentals/master branch in DataApi project. All the code is in fundmantals-loader project in Company model & repository.

And I don't like that I can't define Reader for company entity in terms of datomisca EntityReader[_]. If you'll go to CompanyExplorer in DatomicCompanyRepositoryModule.scala you'll see what I'm talking about. I need to read all the parts of parent object (Company) and then add to implicit scope some readers with predefined child objects. It looks quite ugly.

          val xref = fromEntity[Company.Xref](db.entity(eid))
          val availableInfo = fromEntity[Company.AvailableInfo](db.entity(eid))
          val filingStatus = fromEntity[Company.FilingStatus](db.entity(eid))
          val currencies = fromEntity[Company.Currencies](db.entity(eid))
          val trbc = fromEntity[Option[Company.IndustryClassification]](db.entity(eid))(trbcReader)
          val naics = fromEntity[Option[Company.IndustryClassification]](db.entity(eid))(naics1997Reader)
          val sic = fromEntity[Option[Company.IndustryClassification]](db.entity(eid))(sic1987Reader)
          val industryClassifications = Company.IndustryClassifications(trbc, naics, sic)

          implicit val readStatus = statusReader(availableInfo, filingStatus)
          val status = fromEntity[Company.Status](db.entity(eid))
          implicit val readCompany = companyReader(xref, status, currencies, industryClassifications)

          fromEntity[Company](db.entity(eid))

However I think that my approach of loading data into Datomic as a one time dump of rich domain model violates the idea of data as facts addition/retraction, so I think that I'll try some another approach tomorrow, however the problem of writing custom Readers will still be relevant.

If I would use SBinary idioms, I think the code might look like this:

    implicit object companyStatusFormat extends Format[Company.Status] {
      def reads(in: Input): Company.Status = ....
      def writes(out: Output, value: Company.Statu) = ... 
    }

  implicit object companyFormat extends Format[Company] {
   def reads(in: Input) = Company(
      name = in.reads[Sring](name),
      status = in.reads[Company.Status] 
   ) 
  }

and I will only need single Format[Company] to read all the company model. Also I'm thinking about providing different Format types:

• lazy loading - load only id of referenced entities
• eager - follow reference id to load all domain model object

right now eager behaviour requires to much explicit and knowledge how to load specific objects from Datomic

It's only a proposal, I'm playing with Datomic only second day and still don't have clear understanding how to use it best ;)

[mandubian]

actually I think you can do better with our API : it should be almost as declarative as sbinary style with the power of functional composition. I'll try to find a few minutes tomorrow to write a sample (it's hackday here)

[ezhulenev]

I found declarative solution, thank's to @dwhjames. So issue can be closed ;)

It's a bit verbose, will try to make it beautiful tomorrow.

  implicit val statusReader = new EntityReader[Company.Status] {
    case class Fields(active: Boolean, currentEstimates: Boolean, companyType: String, hasPublicDebt: Boolean)

    implicit val fieldsReader = (
      active.read[Boolean] and
      currentEstimates.read[Boolean] and
        companyType.read[String] and
        hasPublicDebt.read[Boolean]
      )(Fields)

    def read(e: DEntity): Status = {
      val fields = DatomicMapping.fromEntity[Fields](e)
      val availableInfo = DatomicMapping.fromEntity[Company.AvailableInfo](e)
      val filingStatus = DatomicMapping.fromEntity[Company.FilingStatus](e)

      Company.Status(fields.active, fields.currentEstimates, fields.companyType, availableInfo, filingStatus, fields.hasPublicDebt)
    }
  }