Reload analysis files when they change

[Alexey-NM]

It will be very useful to have the ability to refresh bloop without restarting. Would it be possible to add a bloop command for that purpose, it could be something like 'refresh project'

It is very important for large project to have the ability reuse CI compilation cache. jvican has writen: any tool that wants remote compilation can implement it outside of bloop

Unfrotunatly if we replace analysis files the bloop server just will not be able to see changes, We have to stop\start bloop server manually. It is very inconvenient.

[jvican]

Thanks for filing this ticket! Let me get back to this in a few days when I have more bandwidth and we can discuss implementing something like this. Also, if you're building some solution in this space, I suggest you open-source it, it'd be great to have this capability available for folks already using Bloop in their company.

[jvican]

@Alexey-NM Have you made progress writing a solution for semi-remote compilation on your own?

[Alexey-NM]

Currently we are waiting for more stable work of bsp in idea(for exalmpe SCL-16239) We plan to make our own build tool in our application server. It has sense because we have specific development process. So the replication of code is deeply integrated in our application server. Currently I just do not know how to make this code available for other sbt users.

[Alexey-NM]

I have tried to continue working on remote compilation, I created a prototype project on github. https://github.com/global-system/remotebloop

Could you answer? How is it possible to create internal classes manually on load remote state(bloop-internal-classes\bloop-cli-1AkgQmagRb2MnKRiNghfIQ==)?

I do not know whether it will be enough just to replace analysis.

Is it posbile to avoid duplication of classes in bloop-internal-classes and bloop-bsp-clients-classes. if it is possible what is the best way to understand a directory of latest version?

[jvican]

Hey @Alexey-NM, sorry for my late reply. Have you read https://github.com/scalacenter/bloop/issues/1197 ? I need to add this information to the docs website and I will do that shortly. I'm already working on several important changes there.

That link will give you valuable information to get bloop working remotely. Feel free to ask more questions here.

[Alexey-NM]

Feel free to ask more questions here.

Thank you, I plan to finish prototype in near time then I will prepare questions and desirable features.

Have you read #1197 ?

I see the main concept. We have a large project currently it has 600 mega bytes of class files in out folder( it is folder for IntelliJ-BSP) and plus that I has more than 1 giga bytes in ".bloop" folders. Every reboot of bloop(restart computer) always copy all class files in another folder, the class files can leak. Such gready treatment with disk scares. It is very good to have ability to work different client simultaneously, but I feel a lack of exclusive mode. I estimate the end size of our project about 1,3 gigabytes of class files.

[Alexey-NM]

I have created the prototype of utility to save\restore compilation state. See: https://github.com/global-system/remotebloop/blob/master/README.md

Could you answer:

• Is it correct to return emptyStamp in mapBinaryStamp?
• How is it possible to get correct correct hashStamp, "Stamper.forHash(path.toFile)" returns values("57758149f008ab1dedd75a41e225a7066aab5104") wich differ from the values located in analysis("1629395324").

[Alexey-NM]

discuss implementing something like this

I need the ability to flush bloop state to disk and lock it until it is saved\restored. It seems that stopping bloop server is not the good choice(see also long reconnection ) It is very desirable that such mechanism should not require long connection and large copying of files.

It would be good to have the ability to get current "analysisContents.getMiniSetup.output().getSingleOutput" without full loading of analysis for restoring.

[jvican]

Hi @Alexey-NM, I'm glad you're looking into this. I have some reservations about how you're planning to implement remoting. Let me leave you my thoughts.

The bloop state should only be managed by bloop, it is transactional and it defines the core of bloop. The bloop state is defined by all of the output files written to disk and all of the in-memory state. No other tool should use or spy on the bloop state to do X or Y because that can compromise the internal state and it relies on implementation details that certainly will change.

If you need to save the bloop state, I suggest we have a discussion about what exactly you need and we look for ways we can merge that inside bloop itself. But depending on zinc to read analysis files and stamps and using the output files to introspect and manage the bloop state is an approach meant to fail and I'm afraid it's not the way to go.

It is very desirable that such mechanism should not require long connection and large copying of files.

Re class files, copying 1.3 GB of files is nowadays negligible. Bloop has been run in larger codebases than yours without a problem. A normal SSD has a speed of ~1GB/s and keep in mind that the copying is done in parallel. Optimizing the copying of class files is not something on the table for now. In the future, that might change but right now there are other priorities.

If you're actively looking for a way you can run bloop remotely, I suggest we schedule a video conference to chat over the potential approaches and rethink what features should go where and how. That will allow me to advise you about the best way forward and hopefully will speed you up. What do you think?

[olafurpg]

I want to echo @Alexey-NM's concern with classfile copying. I would also love to have an option to disable it. In our case, we would be willing to accept the risks by disabling this behavior.

[jvican]

I would also love to have an option to disable it. In our case, we would be willing to accept the risks by disabling this behavior.

Disabling class file copying is just not possible at all. It can be optimized. If it becomes a priority I'm happy to start a discussion around it, but I think there are more pressing things to do atm.

[Alexey-NM]

If you need to save the bloop state

I agree it is a very broad term.

I suggest we have a discussion about what exactly you need and we look for ways we can merge that inside bloop itself. But depending on zinc to read analysis files and stamps and using the output files to introspect and manage the bloop state is an approach meant to fail and I'm afraid it's not the way to go.

This experiment is based on: https://github.com/sbt/zinc/blob/9200d1e7ec363fc069aa0fc903f548b786410706/internal/zinc-persist/src/main/scala/sbt/internal/inc/cached/ExportableCache.scala

Are there any other alternatives to prevent bloop from full recompilation of a project?

[olafurpg]

I agree disabling classfile copying is not a priority. I was just adding an experience report that we have encountered difficult-to-reproduce problems with this feature when compiling large codebases. We can discuss it separately if it becomes a blocking issue.

[Alexey-NM]

But depending on zinc to read analysis files and stamps and using the output files to introspect and manage the bloop state is an approach meant to fail and I'm afraid it's not the way to go.

I agree that it would be better to leave it as internals to decrease dependency. I can only see one decision, it is to make mapping of analysis internally in bloop and give out some portable artifacts. What can I help to define a roadmap?

[Alexey-NM]

Re class files, copying 1.3 GB of files is nowadays negligible.

It seems it has been mistake to estimate overall size, actually it is much important to consider the amount of files because file opening is a bottleneck, at list in some desctops. I agree it is not priority at any case.