scala-reflect scope

[stanch]

Hi,

I think the scope of "org.scala-lang.macro-paradise" % "scala-reflect" % _ could/should be changed to provided. With that, I am still able to use my project in other places, however without adding a macro-paradise dependency. Do you think there could be any issues with that?

Kind regards, Nick

[xeno-by]

Could you please explain what would be the effect of that change? I have to admit that I'm quite unfamiliar with SBT.

[stanch]

Here is some info about the provided scope (sbt is similar). tl;dr: a dependency marked as provided is used for compilation only, and it’s the runtime’s responsibility to provide it later. So if project A has a provided dependency X, and project B has A in its libraryDependencies, project B will need to include X for A to work. This is usually done for heavy runtimes, such as Akka, or modular components, e.g. different slf4j backends, so that they are not included several times.

As I understand, projects (in this case, “B”), that are using things compiled with paradise210 magic (in this case, “A”) don’t really need scala-reflect at runtime (please correct me, if I am wrong, but apparently it runs fine in my case). However, with the current layout they try to fetch it anyway. Somehow the resolver for macro-paradise is not propagated to B, so everything fails. Adding a resolver to B is easy, but does not feel right (why would one include dangerous things from macro-paradise in a normal project? ;p). So I suggest that this scala-reflect dependency is marked as provided and then just ignored in all Bs to come.

Hope it makes things clearer :)

Kind regards, Nick

[stanch]

Ok, actually in a different project it says

[error] bad symbolic reference. A signature in Lens.class refers to type Aliases
[error] in package scala.reflect.macros which is not available.
[error] It may be completely missing from the current classpath, or the version on
[error] the classpath might be incompatible with the version used when compiling Lens.class.
[error] one error found

So maybe this is not the way to go. Also, it seems that some snapshots, e.g. scala-pickling, scala-reflect, scala-library;2.10.2-SNAPSHOT sometimes go offline (I am not anymore sure, that the observed failure to fetch scala-reflect was due to the missing resolver). Maybe it’s the normal behavior.

[stanch]

Hi,

Unfortunately since yesterday I am no longer able to rebuild neither my project, nor this sample (sbt-example-paradise210). Here’s the output from sbt:

[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.scala-lang#scala-library;2.10.2-SNAPSHOT: not found
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::

I am not sure why everything depends on the vanilla 2.10.2-SNAPSHOT, but it’s indeed missing from sonatype: https://oss.sonatype.org/content/repositories/snapshots/org/scala-lang/scala-library/. Is there something that could be done?

I guess you can disregard anything that I’ve written before, since this may be the root cause.

Kind regards, Nick

[xeno-by]

I'm having the same problem as you, so I don't think it's provided that's at fault. Instead of trying to figure out how SBT works and why it requires vanilla Scala, I'll just roll a 2.10.3-SNAPSHOT of paradise 2.10 in a couple hours. I've already rebased it against 2.10.x (which corresponds to 2.10.3-SNAPSHOT), so we just need to wait for Jenkins to build and publish the artifact.

Since 2.10.x is fully backward and forward compatible, there shouldn't be an issue with using a 2.10.3-SNAPSHOT to build your macros and then running these macros with even 2.10.0, so the change in the version string of paradise 2.10 shouldn't make any difference to your workflow.

[xeno-by]

Indeed, you shouldn't need scala-reflect to use macros if you've already compiled them. Therefore, scala-reflect could be a provided dependency (from what I understand about provided dependencies), and it usually works like that, except for one thing that most probably is a problem in your case.

Here's what happens. scalac stores Scala-specific metadata about classes, members, signatures, etc in an annotation on the containing top-level class. So if you have a top-level object called Lens, then information about all its methods is going to be stuffed into an annotation on this class.

What information I'm talking about? Mostly types – parameter types, member types, base types of classes, etc. Therefore metadata for Lens will include metadata for its macro impls, which will refer to scala-reflect.

Now when you use any method from Lens (or any of its inner classes), scalac will unpickle all the metadata stored in the annotation we were talking about. Not only the metadata that is actually required to typecheck the call, but everything that's in the annotation. And that is going to include information about macro impls, which will require scala-reflect otherwise leading to bad symbolic reference errors. Note that the problem here is not in macro defs - these are fine, because they don't depend on scala-reflect, but rather in macro impls that do.

Bottom line: don't declare macro implementations alongside regular methods, and then it should be possible to use scala-reflect as a provided dependency. (If it doesn't work like that, it's a bug, and please let me know about that).

[xeno-by]

Fixed the problem with paradise 2.10. Please use the updated build config at https://github.com/scalamacros/sbt-example-paradise210/blob/master/project/Build.scala.

[stanch]

That worked out great, thanks! Could you elaborate a bit on “don't declare macro implementations alongside regular methods”? My impls are actually in the companion object, which seems like a natural choice. I will close the issue, since everything builds fine, and I am quite happy with that :)

[xeno-by]

If I recall correctly, companions (classes + objects) are unpickled together, therefore putting a macro impl into an object will prevent its companion class from being used without scala-reflect.

[xeno-by]

/cc @harrah Should I change my macro project examples to use provided?

[harrah]

@xeno-by I would not make it provided by default. My understanding from this thread is that you need scala-reflect in order to compile against a macro-based library. provided would mean that scala-reflect is only used for compiling the macro-based library itself, but not for anyone compiling against the macro-based library. sbt uses Maven metadata for dependency management (pom.xml) and that doesn't have the concept that is necessary here.

[xeno-by]

@harrah If you just use macros from the macro library, then you don't need scala-reflect. If you use macro impls from that library (e.g. to write your own macros), then you do need scala-reflect, but that's natural. Would this fact change your answer?

[harrah]

Isn't that only if you are rather careful and know enough about scalac's unpickling/pickling to avoid the need for scalac to unpickle symbols from scala-reflect?

[xeno-by]

@harrah Yes it is. But suppose that you are careful, then you can use provided, right?

[harrah]

Yes. I wouldn't assume careful by default in an example, though.

[stanch]

Hi, @xeno-by, @harrah!

Here is another tightly related trick question (please let me know if this should be a separate issue). I have a project A with macros (using macro-paradise branch of the library) and project B, which is an Android project that depends on A. Now, B seems to fetch scala-library;2.10.3-SNAPSHOT as a dependency. Android plugin does not like this at all, namely at the ProGuarding step, complaining about duplicated scala libraries. What do you think of the following hack in B? Is it going to work?

libraryDependencies += ("org.foo" %% "bar" % "1.0") exclude ("org.scala-lang.macro-paradise", "scala-library")

I am able to compile and run the project on the phone, but there are no macros yet, so I would like to know in advance if this kind of thing is safe.

Edit: maybe there is a way to mark scala-library as provided in A? I would not say that is a better hack though.

Kind regards, Nick

[harrah]

Is there a reason that you can't use scalaOrganization := "org.scala-lang.macro-paradise" everywhere? I would stick to one and keep it if possible. However, if you only use macros and are careful, yes, you should be fine excluding macro-paradise and using the standard distribution.

[stanch]

@harrah Well, my macro project is a DSL for building Android layouts, so hopefully there’ll be users besides me. I am not sure what is better from the user standpoint. I will definitely follow your suggestion if anything breaks, don’t know why I haven’t thought of it. Anyway, once 2.11 is out (quite soon, as I understand), things will get much better!

[stanch]

Hi,

I am really pleased with today’s stuff! I suppose this repo is rendered obsolete? On the other hand, [1, 2] confuses me. The new setup does not look much different...

[1] https://github.com/scalamacros/sbt-example-paradise/blob/master/project/Build.scala#L6 [2] https://github.com/scalamacros/sbt-example-paradise/blob/master/project/Build.scala#L30

Nick

[xeno-by]

Yes, that's right. This repo is now obsolete. Thanks for pointing out the confusion - updating the readme's of obsoleted paradise examples is something I forgot to do. Now that's fixed.

Indeed the new setup only differs in swapping a custom scala organization for a compiler plugin, but since the entire release thing was quite big (changes to paradise were drastic, I also had to do a major update to the docs, etc), I just decided to start with a clean slate.

Did that bring problems to you? Can I help fixing them?

[stanch]

swapping a custom scala organization for a compiler plugin

But that’s something I was referring to: should we remove organization := "org.scala-lang.macroparadise" and libraryDependencies <+= (scalaVersion)("org.scala-lang" % "scala-reflect" % _)) or not? :)

I understand the scale of changes, of course. Actually I was quite impressed to see that the entire docs updated in one go!

The problems are not there yet—I haven’t started porting things, but will do soon. I can’t promise immediate feedback, because I’m travelling since tomorrow, but on the other hand I’ll be quite probably passing through Lausanne in a couple of days, so if you like I can share some thoughts on my month in macro-paradise :)

[xeno-by]

From what I understand about SBT, organization is only cosmetic, so probably it can be removed. However a dependency on scala-reflect in the macros project is essential, because one cannot define a macro without scala-reflect.jar on the classpath (scala.reflect.macros.Context comes from there).

If you're going to be in Lausanne, I'm absolutely up for meeting. Please let me know if you'll manage to drop by. We can either meet at EPFL or somewhere in the city center. Oh, Ouchy is a nice place as well.

[stanch]

From what I understand about SBT, organization is only cosmetic, so probably it can be removed.

My experience has been that it forces SBT to fetch two scala-libraries and loads of other stuff. Think Android... ouch! scala-reflect is no surprise, that’s fine.

If you're going to be in Lausanne, I'm absolutely up for meeting.

Great! I am flying to Geneva tomorrow and then promptly moving to Montreux the same day. And from there I think we’ll make a trip on 7th or 8th. If that’s ok, I’ll drop you an e-mail once the plan is fully baked.

Nick

[cvogt]

@xeno-by I just found this page googling for "bad symbolic reference. A signature in" "refers to type Aliases". Thx for the explanation not to put macros in companion objects to avoid unpickling issues. It fixed the problem. Maybe you should place this in a scala macros FAQ or something like that. Common trap I suppose.

[xeno-by]

@cvogt Thanks, I'll try to come up with an automatic validator by 2.11.0-RC1.

[xeno-by]

https://issues.scala-lang.org/browse/SI-8090