Closed zekefast closed 8 years ago
Can you send also the code around where this exception is thrown and also how you deploy your application to spark.
On Thu, Apr 21, 2016, 1:09 PM Zeke Fast notifications@github.com wrote:
Overview
I tried to google and investigate this issues by myself, but that not lead me to anything reasonable. Also I quite new to Scala and Spark, so it could be something dummy, but very appreciate any help!
I tried to generate wrappers using ScalaPB for cake protobuf definitions https://github.com/ad2games/cakeproto.
Everything seems working perfectly except one thing. When I tried to access data using cakeproto.affiliates.Affiliate class for affiliate https://github.com/ad2games/cakeproto/blob/c5844813ea85e7d20195578c353d8232e75a28a5/affiliates.proto it gives me follow error.
info] java.lang.ClassNotFoundException: cakeproto.affiliates.Affiliate.PaymentType [info] at java.net.URLClassLoader.findClass(URLClassLoader.java:381) [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) [info] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) [info] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) [info] at java.lang.Class.forName0(Native Method) [info] at java.lang.Class.forName(Class.java:348) [info] at org.apache.spark.util.Utils$.classForName(Utils.scala:174) [info] at org.apache.spark.sql.catalyst.ScalaReflection$.org$apache$spark$sql$catalyst$ScalaReflection$$extractorFor(ScalaReflection.scala:477) [info] at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$9.apply(ScalaReflection.scala:494) [info] at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$9.apply(ScalaReflection.scala:490)
In a short it can not find nested cakeproto.affiliates.Affiliate.PaymentType somehow. Details
Here is generated Affiliate wrapper .
// Generated by the Scala Plugin for the Protocol Buffer Compiler.// Do not edit!//// Protofile syntax: PROTO2 package cakeproto.affiliates
@SerialVersionUID(0L)final case class Affiliate( id: Option[Long] = None, name: Option[String] = None, tier: Option[cakeproto.common.IdName] = None, accountManager: Option[cakeproto.common.IdName] = None, accountStatus: Option[cakeproto.common.IdName] = None, street1: Option[String] = None, street2: Option[String] = None, city: Option[String] = None, state: Option[String] = None, zipCode: Option[String] = None, country: Option[String] = None, website: Option[String] = None, paymentType: Option[cakeproto.affiliates.Affiliate.PaymentType] = None, contacts: Seq[cakeproto.common.Contact] = Nil, tags: Seq[cakeproto.common.IdName] = Nil, trafficPriceFormats: Seq[cakeproto.common.IdName] = Nil, trafficMediaTypes: Seq[cakeproto.common.MediaType] = Nil, trafficVerticalCategories: Seq[cakeproto.common.IdName] = Nil, trafficCountries: Seq[cakeproto.common.Country] = Nil, minimumPaymentThreshold: Option[Double] = None, autoPaymentFee: Option[Double] = None, paymentTo: Option[String] = None, taxClass: Option[String] = None, ssnTaxId: Option[String] = None, payVat: Option[Boolean] = None, swiftIban: Option[String] = None, referralsEnabled: Option[Boolean] = None, referredByAffiliate: Option[cakeproto.common.IdName] = None, referralInfo: Option[String] = None, billingCycle: Option[cakeproto.common.IdName] = None, currency: Option[cakeproto.common.Currency] = None, currencyPaymentSetting: Option[cakeproto.common.IdName] = None, onlineSignup: Option[Boolean] = None, payForConversions: Option[Boolean] = None, review: Option[Boolean] = None, reviewNewSubaffiliates: Option[Boolean] = None, suppression: Option[Boolean] = None, pixelInfo: Option[cakeproto.common.PixelInfo] = None, fireGlobalPixel: Option[Boolean] = None, blacklists: Seq[cakeproto.common.Blacklist] = Nil, redirectDomainOverride: Option[String] = None, autoApproveCampaigns: Option[Boolean] = None, autoApprovePixels: Option[Boolean] = None, hideOffers: Option[Boolean] = None, dateCreated: Option[cakeproto.common.TimeWithZone] = None, notes: Option[String] = None ) extends com.trueaccord.scalapb.GeneratedMessage with com.trueaccord.scalapb.Message[Affiliate] with com.trueaccord.lenses.Updatable[Affiliate] { @transient lazy val serializedSize: Int = { var **size = 0 if (id.isDefined) { size += com.google.protobuf.CodedOutputStream.computeInt64Size(1, id.get) } if (name.isDefined) { size += com.google.protobuf.CodedOutputStream.computeStringSize(2, name.get) } if (tier.isDefined) { size += 1 + com.google.protobuf.CodedOutputStream.computeRawVarint32Size(tier.get.serializedSize) + tier.get.serializedSize } if (accountManager.isDefined) { size += 1 + com.google.protobuf.CodedOutputStream.computeRawVarint32Size(accountManager.get.serializedSize) + accountManager.get.serializedSize } if (accountStatus.isDefined) { size += 1 + com.google.protobuf.CodedOutputStream.computeRawVarint32Size(accountStatus.get.serializedSize) + accountStatus.get.serializedSize } if (street1.isDefined) { size += com.google.protobuf.CodedOutputStream.computeStringSize(6, street1.get) } if (street2.isDefined) { size += com.google.protobuf.CodedOutputStream.computeStringSize(7, street2.get) } if (city.isDefined) { size += com.google.protobuf.CodedOutputStream.computeStringSize(8, city.get) } if (state.isDefined) { size += com.google.protobuf.CodedOutputStream.computeStringSize(9, state.get) } if (zipCode.isDefined) { size += com.google.protobuf.CodedOutputStream.computeStringSize(10, zipCode.get) } if (country.isDefined) { size += com.google.protobuf.CodedOutputStream.computeStringSize(11, country.get) } if (website.isDefined) { size += com.google.protobuf.CodedOutputStream.computeStringSize(12, website.get) } if (paymentType.isDefined) { size += 1 + com.google.protobuf.CodedOutputStream.computeRawVarint32Size(paymentType.get.serializedSize) + paymentType.get.serializedSize } contacts.foreach(contacts => size += 1 + com.google.protobuf.CodedOutputStream.computeRawVarint32Size(contacts.serializedSize) + contacts.serializedSize) tags.foreach(tags => size += 1 + com.google.protobuf.CodedOutputStream.computeRawVarint32Size(tags.serializedSize) + tags.serializedSize) trafficPriceFormats.foreach(trafficPriceFormats => size += 2 + com.google.protobuf.CodedOutputStream.computeRawVarint32Size(trafficPriceFormats.serializedSize) + trafficPriceFormats.serializedSize) trafficMediaTypes.foreach(trafficMediaTypes => size += 2 + com.google.protobuf.CodedOutputStream.computeRawVarint32Size(trafficMediaTypes.serializedSize) + trafficMediaTypes.serializedSize) trafficVerticalCategories.foreach(trafficVerticalCategories => size += 2 + com.google.protobuf.CodedOutputStream.computeRawVarint32Size(trafficVerticalCategories.serializedSize) + trafficVerticalCategories.serializedSize) trafficCountries.foreach(trafficCountries => size += 2 + com.google.protobuf.CodedOutputStream.computeRawVarint32Size(trafficCountries.serializedSize) + trafficCountries.serializedSize) if (minimumPaymentThreshold.isDefined) { size += com.google.protobuf.CodedOutputStream.computeDoubleSize(20, minimumPaymentThreshold.get) } if (autoPaymentFee.isDefined) { size += com.google.protobuf.CodedOutputStream.computeDoubleSize(21, autoPaymentFee.get) } if (paymentTo.isDefined) { size += com.google.protobuf.CodedOutputStream.computeStringSize(22, paymentTo.get) } if (taxClass.isDefined) { size += com.google.protobuf.CodedOutputStream.computeStringSize(23, taxClass.get) } if (ssnTaxId.isDefined) { size += com.google.protobuf.CodedOutputStream.computeStringSize(24, ssnTaxId.get) } if (payVat.isDefined) { size += com.google.protobuf.CodedOutputStream.computeBoolSize(25, payVat.get) } if (swiftIban.isDefined) { size += com.google.protobuf.CodedOutputStream.computeStringSize(26, swiftIban.get) } if (referralsEnabled.isDefined) { size += com.google.protobuf.CodedOutputStream.computeBoolSize(27, referralsEnabled.get) } if (referredByAffiliate.isDefined) { size += 2 + com.google.protobuf.CodedOutputStream.computeRawVarint32Size(referredByAffiliate.get.serializedSize) + referredByAffiliate.get.serializedSize } if (referralInfo.isDefined) { size += com.google.protobuf.CodedOutputStream.computeStringSize(29, referralInfo.get) } if (billingCycle.isDefined) { size += 2 + com.google.protobuf.CodedOutputStream.computeRawVarint32Size(billingCycle.get.serializedSize) + billingCycle.get.serializedSize } if (currency.isDefined) { size += 2 + com.google.protobuf.CodedOutputStream.computeRawVarint32Size(currency.get.serializedSize) + currency.get.serializedSize } if (currencyPaymentSetting.isDefined) { size += 2 + com.google.protobuf.CodedOutputStream.computeRawVarint32Size(currencyPaymentSetting.get.serializedSize) + currencyPaymentSetting.get.serializedSize } if (onlineSignup.isDefined) { size += com.google.protobuf.CodedOutputStream.computeBoolSize(33, onlineSignup.get) } if (payForConversions.isDefined) { size += com.google.protobuf.CodedOutputStream.computeBoolSize(34, payForConversions.get) } if (review.isDefined) { size += com.google.protobuf.CodedOutputStream.computeBoolSize(35, review.get) } if (reviewNewSubaffiliates.isDefined) { size += com.google.protobuf.CodedOutputStream.computeBoolSize(36, reviewNewSubaffiliates.get) } if (suppression.isDefined) { size += com.google.protobuf.CodedOutputStream.computeBoolSize(37, suppression.get) } if (pixelInfo.isDefined) { size += 2 + com.google.protobuf.CodedOutputStream.computeRawVarint32Size(pixelInfo.get.serializedSize) + pixelInfo.get.serializedSize } if (fireGlobalPixel.isDefined) { size += com.google.protobuf.CodedOutputStream.computeBoolSize(39, fireGlobalPixel.get) } blacklists.foreach(blacklists => size += 2 + com.google.protobuf.CodedOutputStream.computeRawVarint32Size(blacklists.serializedSize) + blacklists.serializedSize) if (redirectDomainOverride.isDefined) { size += com.google.protobuf.CodedOutputStream.computeStringSize(41, redirectDomainOverride.get) } if (autoApproveCampaigns.isDefined) { size += com.google.protobuf.CodedOutputStream.computeBoolSize(42, autoApproveCampaigns.get) } if (autoApprovePixels.isDefined) { size += com.google.protobuf.CodedOutputStream.computeBoolSize(43, autoApprovePixels.get) } if (hideOffers.isDefined) { size += com.google.protobuf.CodedOutputStream.computeBoolSize(44, hideOffers.get) } if (dateCreated.isDefined) { size += 2 + com.google.protobuf.CodedOutputStream.computeRawVarint32Size(dateCreated.get.serializedSize) + dateCreated.get.serializedSize } if (notes.isDefined) { size += com.google.protobuf.CodedOutputStream.computeStringSize(46, notes.get) } size } def writeTo(output: com.google.protobuf.CodedOutputStream): Unit = { id.foreach { v => output.writeInt64(1, v) }; name.foreach { v => output.writeString(2, v) }; tier.foreach { v => output.writeTag(3, 2) output.writeRawVarint32(v.serializedSize) v.writeTo(output) }; accountManager.foreach { v => output.writeTag(4, 2) output.writeRawVarint32(v.serializedSize) v.writeTo(output) }; accountStatus.foreach { v => output.writeTag(5, 2) output.writeRawVarint32(v.serializedSize) v.writeTo(output) }; street1.foreach { v => output.writeString(6, v) }; street2.foreach { v => output.writeString(7, v) }; city.foreach { v => output.writeString(8, v) }; state.foreach { v => output.writeString(9, v) }; zipCode.foreach { v => output.writeString(10, v) }; country.foreach { v => output.writeString(11, v) }; website.foreach { v => output.writeString(12, v) }; paymentType.foreach { v => output.writeTag(13, 2) output.writeRawVarint32(v.serializedSize) v.writeTo(output) }; contacts.foreach { v => output.writeTag(14, 2) output.writeRawVarint32(v.serializedSize) v.writeTo(output) }; tags.foreach { v => output.writeTag(15, 2) output.writeRawVarint32(v.serializedSize) v.writeTo(output) }; trafficPriceFormats.foreach { v => output.writeTag(16, 2) output.writeRawVarint32(v.serializedSize) v.writeTo(output) }; trafficMediaTypes.foreach { v => output.writeTag(17, 2) output.writeRawVarint32(v.serializedSize) v.writeTo(output) }; trafficVerticalCategories.foreach { v => output.writeTag(18, 2) output.writeRawVarint32(v.serializedSize) v.writeTo(output) }; trafficCountries.foreach { v => output.writeTag(19, 2) output.writeRawVarint32(v.serializedSize) v.writeTo(output) }; minimumPaymentThreshold.foreach { v => output.writeDouble(20, v) }; autoPaymentFee.foreach { v => output.writeDouble(21, v) }; paymentTo.foreach { v => output.writeString(22, v) }; taxClass.foreach { v => output.writeString(23, v) }; ssnTaxId.foreach { v => output.writeString(24, v) }; payVat.foreach { v => output.writeBool(25, v) }; swiftIban.foreach { v => output.writeString(26, v) }; referralsEnabled.foreach { v => output.writeBool(27, v) }; referredByAffiliate.foreach { v => output.writeTag(28, 2) output.writeRawVarint32(v.serializedSize) v.writeTo(output) }; referralInfo.foreach { v => output.writeString(29, v) }; billingCycle.foreach { v => output.writeTag(30, 2) output.writeRawVarint32(v.serializedSize) v.writeTo(output) }; currency.foreach { v => output.writeTag(31, 2) output.writeRawVarint32(v.serializedSize) v.writeTo(output) }; currencyPaymentSetting.foreach { v => output.writeTag(32, 2) output.writeRawVarint32(v.serializedSize) v.writeTo(output) }; onlineSignup.foreach { v => output.writeBool(33, v) }; payForConversions.foreach { v => output.writeBool(34, v) }; review.foreach { v => output.writeBool(35, v) }; reviewNewSubaffiliates.foreach { v => output.writeBool(36, v) }; suppression.foreach { v => output.writeBool(37, v) }; pixelInfo.foreach { v => output.writeTag(38, 2) output.writeRawVarint32(v.serializedSize) v.writeTo(output) }; fireGlobalPixel.foreach { v => output.writeBool(39, v) }; blacklists.foreach { v => output.writeTag(40, 2) output.writeRawVarint32(v.serializedSize) v.writeTo(output) }; redirectDomainOverride.foreach { v => output.writeString(41, v) }; autoApproveCampaigns.foreach { v => output.writeBool(42, v) }; autoApprovePixels.foreach { v => output.writeBool(43, v) }; hideOffers.foreach { v => output.writeBool(44, v) }; dateCreated.foreach { v => output.writeTag(45, 2) output.writeRawVarint32(v.serializedSize) v.writeTo(output) }; notes.foreach { v => output.writeString(46, v) }; } def mergeFrom(input: com.google.protobuf.CodedInputStream): cakeproto.affiliates.Affiliate = { var id = this.id var name = this.name var tier = this.tier var accountManager = this.accountManager var accountStatus = this.accountStatus var street1 = this.street1 var street2 = this.street2 var city = this.city var state = this.state var zipCode = this.zipCode var country = this.country var website = this.website var paymentType = this.paymentType val contacts = (scala.collection.immutable.Vector.newBuilder[cakeproto.common.Contact] ++= this.contacts) val tags = (scala.collection.immutable.Vector.newBuilder[cakeproto.common.IdName] ++= this.tags) val trafficPriceFormats = (scala.collection.immutable.Vector.newBuilder[cakeproto.common.IdName] ++= this.trafficPriceFormats) val trafficMediaTypes = (scala.collection.immutable.Vector.newBuilder[cakeproto.common.MediaType] ++= this.trafficMediaTypes) val trafficVerticalCategories = (scala.collection.immutable.Vector.newBuilder[cakeproto.common.IdName] ++= this.trafficVerticalCategories) val trafficCountries = (scala.collection.immutable.Vector.newBuilder[cakeproto.common.Country] ++= this.trafficCountries) var minimumPaymentThreshold = this.minimumPaymentThreshold var autoPaymentFee = this.autoPaymentFee var paymentTo = this.paymentTo var taxClass = this.taxClass var ssnTaxId = this.ssnTaxId var payVat = this.payVat var swiftIban = this.swiftIban var referralsEnabled = this.referralsEnabled var referredByAffiliate = this.referredByAffiliate var referralInfo = this.referralInfo var billingCycle = this.billingCycle var currency = this.currency var currencyPaymentSetting = this.currencyPaymentSetting var onlineSignup = this.onlineSignup var payForConversions = this.payForConversions var review = this.review var reviewNewSubaffiliates = this.reviewNewSubaffiliates var suppression = this.suppression var pixelInfo = this.pixelInfo var fireGlobalPixel = this.fireGlobalPixel val blacklists = (scala.collection.immutable.Vector.newBuilder[cakeproto.common.Blacklist] ++= this.blacklists) var redirectDomainOverride = this.redirectDomainOverride var autoApproveCampaigns = this.autoApproveCampaigns var autoApprovePixels = this.autoApprovePixels var hideOffers = this.hideOffers var dateCreated = this.dateCreated var notes = this.notes var _done** = false while (!_done) { val _tag = input.readTag() _tag match { case 0 => _done = true case 8 => id = Some(input.readInt64()) case 18 => name = Some(input.readString()) case 26 => tier = Some(com.trueaccord.scalapb.LiteParser.readMessage(input, tier.getOrElse(cakeproto.common.IdName.defaultInstance))) case 34 => accountManager = Some(com.trueaccord.scalapb.LiteParser.readMessage(input, accountManager.getOrElse(cakeproto.common.IdName.defaultInstance))) case 42 => accountStatus = Some(com.trueaccord.scalapb.LiteParser.readMessage(input, accountStatus.getOrElse(cakeproto.common.IdName.defaultInstance))) case 50 => street1 = Some(input.readString()) case 58 => street2 = Some(input.readString()) case 66 => city = Some(input.readString()) case 74 => state = Some(input.readString()) case 82 => zipCode = Some(input.readString()) case 90 => country = Some(input.readString()) case 98 => website = Some(input.readString()) case 106 => paymentType = Some(com.trueaccord.scalapb.LiteParser.readMessage(input, paymentType.getOrElse(cakeproto.affiliates.Affiliate.PaymentType.defaultInstance))) case 114 => contacts += com.trueaccord.scalapb.LiteParser.readMessage(input, cakeproto.common.Contact.defaultInstance) case 122 => tags += com.trueaccord.scalapb.LiteParser.readMessage(input, cakeproto.common.IdName.defaultInstance) case 130 => trafficPriceFormats += com.trueaccord.scalapb.LiteParser.readMessage(input, cakeproto.common.IdName.defaultInstance) case 138 => trafficMediaTypes += com.trueaccord.scalapb.LiteParser.readMessage(input, cakeproto.common.MediaType.defaultInstance) case 146 => trafficVerticalCategories += com.trueaccord.scalapb.LiteParser.readMessage(input, cakeproto.common.IdName.defaultInstance) case 154 => trafficCountries += com.trueaccord.scalapb.LiteParser.readMessage(input, cakeproto.common.Country.defaultInstance) case 161 => minimumPaymentThreshold = Some(input.readDouble()) case 169 => autoPaymentFee = Some(input.readDouble()) case 178 => paymentTo = Some(input.readString()) case 186 => taxClass = Some(input.readString()) case 194 => ssnTaxId = Some(input.readString()) case 200 => payVat = Some(input.readBool()) case 210 => swiftIban = Some(input.readString()) case 216 => referralsEnabled = Some(input.readBool()) case 226 => referredByAffiliate = Some(com.trueaccord.scalapb.LiteParser.readMessage(input, referredByAffiliate.getOrElse(cakeproto.common.IdName.defaultInstance))) case 234 => referralInfo = Some(input.readString()) case 242 => billingCycle = Some(com.trueaccord.scalapb.LiteParser.readMessage(input, billingCycle.getOrElse(cakeproto.common.IdName.defaultInstance))) case 250 => currency = Some(com.trueaccord.scalapb.LiteParser.readMessage(input, currency.getOrElse(cakeproto.common.Currency.defaultInstance))) case 258 => currencyPaymentSetting = Some(com.trueaccord.scalapb.LiteParser.readMessage(input, currencyPaymentSetting.getOrElse(cakeproto.common.IdName.defaultInstance))) case 264 => onlineSignup = Some(input.readBool()) case 272 => payForConversions = Some(input.readBool()) case 280 => review = Some(input.readBool()) case 288 => reviewNewSubaffiliates = Some(input.readBool()) case 296 => suppression = Some(input.readBool()) case 306 => pixelInfo = Some(com.trueaccord.scalapb.LiteParser.readMessage(input, pixelInfo.getOrElse(cakeproto.common.PixelInfo.defaultInstance))) case 312 => fireGlobalPixel = Some(input.readBool()) case 322 => blacklists += com.trueaccord.scalapb.LiteParser.readMessage(input, cakeproto.common.Blacklist.defaultInstance) case 330 => redirectDomainOverride = Some(input.readString()) case 336 => autoApproveCampaigns = Some(input.readBool()) case 344 => autoApprovePixels = Some(input.readBool()) case 352 => hideOffers = Some(input.readBool()) case 362 => dateCreated = Some(com.trueaccord.scalapb.LiteParser.readMessage(input, dateCreated.getOrElse(cakeproto.common.TimeWithZone.defaultInstance))) case 370 => notes = Some(input.readString()) case tag => input.skipField(tag) } } cakeproto.affiliates.Affiliate( id = id, name = name, tier = tier, accountManager = accountManager, accountStatus = accountStatus, street1 = street1, street2 = street2, city = city, state = state, zipCode = zipCode, country = country, website = website, paymentType = paymentType, contacts = contacts.result(), tags = tags.result(), trafficPriceFormats = trafficPriceFormats.result(), trafficMediaTypes = trafficMediaTypes.result(), trafficVerticalCategories = trafficVerticalCategories.result(), trafficCountries = trafficCountries.result(), minimumPaymentThreshold = minimumPaymentThreshold, autoPaymentFee = autoPaymentFee, paymentTo = paymentTo, taxClass = taxClass, ssnTaxId = ssnTaxId, payVat = payVat, swiftIban = swiftIban, referralsEnabled = referralsEnabled, referredByAffiliate = referredByAffiliate, referralInfo = referralInfo, billingCycle = billingCycle, currency = currency, currencyPaymentSetting = currencyPaymentSetting, onlineSignup = onlineSignup, payForConversions = payForConversions, review = review, reviewNewSubaffiliates = reviewNewSubaffiliates, suppression = suppression, pixelInfo = pixelInfo, fireGlobalPixel = fireGlobalPixel, blacklists = blacklists.result(), redirectDomainOverride = redirectDomainOverride, autoApproveCampaigns = autoApproveCampaigns, autoApprovePixels = autoApprovePixels, hideOffers = hideOffers, dateCreated = dateCreated, notes = notes ) } def getId: Long = id.getOrElse(0L) def clearId: Affiliate = copy(id = None) def withId(v: Long): Affiliate = copy(id = Some(v)) def getName: String = name.getOrElse("") def clearName: Affiliate = copy(name = None) def withName(v: String): Affiliate = copy(name = Some(v)) def getTier: cakeproto.common.IdName = tier.getOrElse(cakeproto.common.IdName.defaultInstance) def clearTier: Affiliate = copy(tier = None) def withTier(v: cakeproto.common.IdName): Affiliate = copy(tier = Some(v)) def getAccountManager: cakeproto.common.IdName = accountManager.getOrElse(cakeproto.common.IdName.defaultInstance) def clearAccountManager: Affiliate = copy(accountManager = None) def withAccountManager(v: cakeproto.common.IdName): Affiliate = copy(accountManager = Some(v)) def getAccountStatus: cakeproto.common.IdName = accountStatus.getOrElse(cakeproto.common.IdName.defaultInstance) def clearAccountStatus: Affiliate = copy(accountStatus = None) def withAccountStatus(v: cakeproto.common.IdName): Affiliate = copy(accountStatus = Some(v)) def getStreet1: String = street1.getOrElse("") def clearStreet1: Affiliate = copy(street1 = None) def withStreet1(v: String): Affiliate = copy(street1 = Some(v)) def getStreet2: String = street2.getOrElse("") def clearStreet2: Affiliate = copy(street2 = None) def withStreet2(v: String): Affiliate = copy(street2 = Some(v)) def getCity: String = city.getOrElse("") def clearCity: Affiliate = copy(city = None) def withCity(v: String): Affiliate = copy(city = Some(v)) def getState: String = state.getOrElse("") def clearState: Affiliate = copy(state = None) def withState(v: String): Affiliate = copy(state = Some(v)) def getZipCode: String = zipCode.getOrElse("") def clearZipCode: Affiliate = copy(zipCode = None) def withZipCode(v: String): Affiliate = copy(zipCode = Some(v)) def getCountry: String = country.getOrElse("") def clearCountry: Affiliate = copy(country = None) def withCountry(v: String): Affiliate = copy(country = Some(v)) def getWebsite: String = website.getOrElse("") def clearWebsite: Affiliate = copy(website = None) def withWebsite(v: String): Affiliate = copy(website = Some(v)) def getPaymentType: cakeproto.affiliates.Affiliate.PaymentType = paymentType.getOrElse(cakeproto.affiliates.Affiliate.PaymentType.defaultInstance) def clearPaymentType: Affiliate = copy(paymentType = None) def withPaymentType(v: cakeproto.affiliates.Affiliate.PaymentType): Affiliate = copy(paymentType = Some(v)) def clearContacts = copy(contacts = Seq.empty) def addContacts(vs: cakeproto.common.Contact): Affiliate = addAllContacts(vs) def addAllContacts(vs: TraversableOnce[cakeproto.common.Contact]): Affiliate = copy(contacts = contacts ++ vs) def withContacts(v: Seq[cakeproto.common.Contact]): Affiliate = copy(contacts = v) def clearTags = copy(tags = Seq.empty) def addTags(vs: cakeproto.common.IdName): Affiliate = addAllTags(vs) def addAllTags(vs: TraversableOnce[cakeproto.common.IdName]): Affiliate = copy(tags = tags ++ vs) def withTags(v: Seq[cakeproto.common.IdName]): Affiliate = copy(tags = v) def clearTrafficPriceFormats = copy(trafficPriceFormats = Seq.empty) def addTrafficPriceFormats(vs: cakeproto.common.IdName): Affiliate = addAllTrafficPriceFormats(vs) def addAllTrafficPriceFormats(vs: TraversableOnce[cakeproto.common.IdName]): Affiliate = copy(trafficPriceFormats = trafficPriceFormats ++ vs) def withTrafficPriceFormats(v: Seq[cakeproto.common.IdName]): Affiliate = copy(trafficPriceFormats = v) def clearTrafficMediaTypes = copy(trafficMediaTypes = Seq.empty) def addTrafficMediaTypes(vs: cakeproto.common.MediaType): Affiliate = addAllTrafficMediaTypes(vs) def addAllTrafficMediaTypes(vs: TraversableOnce[cakeproto.common.MediaType]): Affiliate = copy(trafficMediaTypes = trafficMediaTypes ++ vs) def withTrafficMediaTypes(v: Seq[cakeproto.common.MediaType]): Affiliate = copy(trafficMediaTypes = v) def clearTrafficVerticalCategories = copy(trafficVerticalCategories = Seq.empty) def addTrafficVerticalCategories(vs: cakeproto.common.IdName): Affiliate = addAllTrafficVerticalCategories(vs) def addAllTrafficVerticalCategories(vs: TraversableOnce[cakeproto.common.IdName]): Affiliate = copy(trafficVerticalCategories = trafficVerticalCategories ++ vs) def withTrafficVerticalCategories(v: Seq[cakeproto.common.IdName]): Affiliate = copy(trafficVerticalCategories = v) def clearTrafficCountries = copy(trafficCountries = Seq.empty) def addTrafficCountries(vs: cakeproto.common.Country): Affiliate = addAllTrafficCountries(vs) def addAllTrafficCountries(vs: TraversableOnce[cakeproto.common.Country]): Affiliate = copy(trafficCountries = trafficCountries ++ vs) def withTrafficCountries(v: Seq[cakeproto.common.Country]): Affiliate = copy(trafficCountries = v) def getMinimumPaymentThreshold: Double = minimumPaymentThreshold.getOrElse(0.0) def clearMinimumPaymentThreshold: Affiliate = copy(minimumPaymentThreshold = None) def withMinimumPaymentThreshold(v: Double): Affiliate = copy(minimumPaymentThreshold = Some(v)) def getAutoPaymentFee: Double = autoPaymentFee.getOrElse(0.0) def clearAutoPaymentFee: Affiliate = copy(autoPaymentFee = None) def withAutoPaymentFee(v: Double): Affiliate = copy(autoPaymentFee = Some(v)) def getPaymentTo: String = paymentTo.getOrElse("") def clearPaymentTo: Affiliate = copy(paymentTo = None) def withPaymentTo(v: String): Affiliate = copy(paymentTo = Some(v)) def getTaxClass: String = taxClass.getOrElse("") def clearTaxClass: Affiliate = copy(taxClass = None) def withTaxClass(v: String): Affiliate = copy(taxClass = Some(v)) def getSsnTaxId: String = ssnTaxId.getOrElse("") def clearSsnTaxId: Affiliate = copy(ssnTaxId = None) def withSsnTaxId(v: String): Affiliate = copy(ssnTaxId = Some(v)) def getPayVat: Boolean = payVat.getOrElse(false) def clearPayVat: Affiliate = copy(payVat = None) def withPayVat(v: Boolean): Affiliate = copy(payVat = Some(v)) def getSwiftIban: String = swiftIban.getOrElse("") def clearSwiftIban: Affiliate = copy(swiftIban = None) def withSwiftIban(v: String): Affiliate = copy(swiftIban = Some(v)) def getReferralsEnabled: Boolean = referralsEnabled.getOrElse(false) def clearReferralsEnabled: Affiliate = copy(referralsEnabled = None) def withReferralsEnabled(v: Boolean): Affiliate = copy(referralsEnabled = Some(v)) def getReferredByAffiliate: cakeproto.common.IdName = referredByAffiliate.getOrElse(cakeproto.common.IdName.defaultInstance) def clearReferredByAffiliate: Affiliate = copy(referredByAffiliate = None) def withReferredByAffiliate(v: cakeproto.common.IdName): Affiliate = copy(referredByAffiliate = Some(v)) def getReferralInfo: String = referralInfo.getOrElse("") def clearReferralInfo: Affiliate = copy(referralInfo = None) def withReferralInfo(v: String): Affiliate = copy(referralInfo = Some(v)) def getBillingCycle: cakeproto.common.IdName = billingCycle.getOrElse(cakeproto.common.IdName.defaultInstance) def clearBillingCycle: Affiliate = copy(billingCycle = None) def withBillingCycle(v: cakeproto.common.IdName): Affiliate = copy(billingCycle = Some(v)) def getCurrency: cakeproto.common.Currency = currency.getOrElse(cakeproto.common.Currency.defaultInstance) def clearCurrency: Affiliate = copy(currency = None) def withCurrency(v: cakeproto.common.Currency): Affiliate = copy(currency = Some(v)) def getCurrencyPaymentSetting: cakeproto.common.IdName = currencyPaymentSetting.getOrElse(cakeproto.common.IdName.defaultInstance) def clearCurrencyPaymentSetting: Affiliate = copy(currencyPaymentSetting = None) def withCurrencyPaymentSetting(v: cakeproto.common.IdName): Affiliate = copy(currencyPaymentSetting = Some(v)) def getOnlineSignup: Boolean = onlineSignup.getOrElse(false) def clearOnlineSignup: Affiliate = copy(onlineSignup = None) def withOnlineSignup(v: Boolean): Affiliate = copy(onlineSignup = Some(v)) def getPayForConversions: Boolean = payForConversions.getOrElse(false) def clearPayForConversions: Affiliate = copy(payForConversions = None) def withPayForConversions(v: Boolean): Affiliate = copy(payForConversions = Some(v)) def getReview: Boolean = review.getOrElse(false) def clearReview: Affiliate = copy(review = None) def withReview(v: Boolean): Affiliate = copy(review = Some(v)) def getReviewNewSubaffiliates: Boolean = reviewNewSubaffiliates.getOrElse(false) def clearReviewNewSubaffiliates: Affiliate = copy(reviewNewSubaffiliates = None) def withReviewNewSubaffiliates(v: Boolean): Affiliate = copy(reviewNewSubaffiliates = Some(v)) def getSuppression: Boolean = suppression.getOrElse(false) def clearSuppression: Affiliate = copy(suppression = None) def withSuppression(v: Boolean): Affiliate = copy(suppression = Some(v)) def getPixelInfo: cakeproto.common.PixelInfo = pixelInfo.getOrElse(cakeproto.common.PixelInfo.defaultInstance) def clearPixelInfo: Affiliate = copy(pixelInfo = None) def withPixelInfo(v: cakeproto.common.PixelInfo): Affiliate = copy(pixelInfo = Some(v)) def getFireGlobalPixel: Boolean = fireGlobalPixel.getOrElse(false) def clearFireGlobalPixel: Affiliate = copy(fireGlobalPixel = None) def withFireGlobalPixel(v: Boolean): Affiliate = copy(fireGlobalPixel = Some(v)) def clearBlacklists = copy(blacklists = Seq.empty) def addBlacklists(vs: cakeproto.common.Blacklist*): Affiliate = addAllBlacklists(vs) def addAllBlacklists(vs: TraversableOnce[cakeproto.common.Blacklist]): Affiliate = copy(blacklists = blacklists ++ vs) def withBlacklists(v: Seq[cakeproto.common.Blacklist]): Affiliate = copy(blacklists = v) def getRedirectDomainOverride: String = redirectDomainOverride.getOrElse("") def clearRedirectDomainOverride: Affiliate = copy(redirectDomainOverride = None) def withRedirectDomainOverride(v: String): Affiliate = copy(redirectDomainOverride = Some(v)) def getAutoApproveCampaigns: Boolean = autoApproveCampaigns.getOrElse(false) def clearAutoApproveCampaigns: Affiliate = copy(autoApproveCampaigns = None) def withAutoApproveCampaigns(v: Boolean): Affiliate = copy(autoApproveCampaigns = Some(v)) def getAutoApprovePixels: Boolean = autoApprovePixels.getOrElse(false) def clearAutoApprovePixels: Affiliate = copy(autoApprovePixels = None) def withAutoApprovePixels(v: Boolean): Affiliate = copy(autoApprovePixels = Some(v)) def getHideOffers: Boolean = hideOffers.getOrElse(false) def clearHideOffers: Affiliate = copy(hideOffers = None) def withHideOffers(v: Boolean): Affiliate = copy(hideOffers = Some(v)) def getDateCreated: cakeproto.common.TimeWithZone = dateCreated.getOrElse(cakeproto.common.TimeWithZone.defaultInstance) def clearDateCreated: Affiliate = copy(dateCreated = None) def withDateCreated(v: cakeproto.common.TimeWithZone): Affiliate = copy(dateCreated = Some(v)) def getNotes: String = notes.getOrElse("") def clearNotes: Affiliate = copy(notes = None) def withNotes(v: String): Affiliate = copy(notes = Some(v)) def getField(field: com.google.protobuf.Descriptors.FieldDescriptor): scala.Any = { __field.getNumber match { case 1 => id.getOrElse(null) case 2 => name.getOrElse(null) case 3 => tier.getOrElse(null) case 4 => accountManager.getOrElse(null) case 5 => accountStatus.getOrElse(null) case 6 => street1.getOrElse(null) case 7 => street2.getOrElse(null) case 8 => city.getOrElse(null) case 9 => state.getOrElse(null) case 10 => zipCode.getOrElse(null) case 11 => country.getOrElse(null) case 12 => website.getOrElse(null) case 13 => paymentType.getOrElse(null) case 14 => contacts case 15 => tags case 16 => trafficPriceFormats case 17 => trafficMediaTypes
Hi @thesamet Thank you for quick response!
In particular, this appears also in tests.
Tests bootstrap SparkContext
:
val conf = new SparkConf().setAppName(getClass.getSimpleName).setMaster("local[2]")
val sc = new SparkContext(conf)
sc.setLogLevel("WARN")
sqlContext = new SQLContext(sc)
val path = getClass().getResource("/cake").toString
dataStore = new CakeDataStore(path)(sqlContext)
Then method on CakeDataStore
is called.
Inside CakeDataStore
we have data loading like follow
val df = sqlContext
.read.format(s"com.ad2games.spark.cakeproto.$tableName")
.load(fullPath)
This is an exact part which is failing I guess.
There is also a bunch of juggling in our cakeproto
library around ProviderRelation
from spark and reader which unzip protobuf data from S3.
Do you want to see them also?
I am not sure about the .read.format()
line. Are you relying somehow on sparksql-scalapb (https://trueaccord.github.io/ScalaPB/sparksql.html) ?
I'll need a minimal project that exhibits this problem. Can you set it up and upload to github?
We do not use spark-scalapb
.
I'll try to prepare test project for that issue.
Overview
I tried to google and investigate this issues by myself, but that not lead me to anything reasonable. Also I quite new to Scala and Spark, so it could be something dummy, but very appreciate any help!
I tried to generate wrappers using
ScalaPB
for cake protobuf definitions.Everything seems working perfectly except one thing. When I tried to access data using
cakeproto.affiliates.Affiliate
class foraffiliate
it gives me follow error.In a short it can not find nested
cakeproto.affiliates.Affiliate.PaymentType
somehow.Details
Here is generated
Affiliate
wrapper .I found a workaround to that problem. When I edit that definition and move nested
PaymentType
class out ofAffiliate
the error is gone.Modified version of
Affiliate.scala
file.Environment
sbt-scalabt
:0.5.25
1.6.1
2.11.7
0.13.11
OpenJDK 64-Bit Server VM, Java 1.8.0_72-internal
2.6.1
(fromprotobuf-compiler
package on Debian Jessie)Question
It would be good to understand the source of this error and have a fix for it. Is there any configuration option to unfold
PaymentType
in generated code or any other ideas how it could be possible fixed?