search

platformio / platformio-core

Your Gateway to Embedded Software Development Excellence :alien:
https://platformio.org
Apache License 2.0
7.93k stars 792 forks source link

Implement "lib_extra_dirs" option for project environment #537

Closed glelouet closed 8 years ago

glelouet commented 8 years ago

My platformio.ini :

[platformio] lib_dir = ../lib

[env:uno] platform = atmelavr

1=onewire

lib_install = 1

I also have ../lib/mylib/lib.h that is #include in my sources when I init, platformio downloads the onewire lib and add it to ../lib . However this leads to two issues

possible solution is to have two environment variables : lib_dir to specify where we PUT the required libs, and inc_dir to specify where are our private library not managed by platformio downloader. This way, I could set inc_dir to ../lib (or ../lib, ./lib, ../../lib with a list) in my platformio.ini , and set the lib_dir in the cli if modification required (eg using arduino IDE)

glelouet commented 8 years ago

PlatformIO, version 2.8.4

ivankravets commented 8 years ago

The lib directory in project is used for the private libraries. PlatformIO Library Manager doesn't touch them. The lib_dir in platformio.ini allows to override default location for the GLOBAL libs.

I want to mention that in PlatformIO 3.0 the logic with Library Manager will be rewritten from the scratch. By default all libs will be installed in project/lib directory with semver support. If you want to install library globally, then need to specify platformio lib install -g/--global flag. In this case, the library will be installed globally and will be accessed for the all projects. See https://github.com/platformio/platformio/issues/475.

I recommend to take a look on the all lib-related issues for PlatformIO 3.0. https://github.com/platformio/platformio/issues?q=is%3Aopen+milestone%3A3.0.0+label%3Alib

The question: will PlatformIO 3.0 resolve your issue?

glelouet commented 8 years ago

The way you explain it, it won't, because my private library is in ../lib and not in ./lib .

In my case I need to have :

If I understood well, the first point is not possible in pio 3 , I assume the semver lib dir can be changed so the second point would be.

ivankravets commented 8 years ago

How about lib_extra_dirs per [env:xxx] environment? Users will have ability to tell PlatformIO Library Dependency Finder extra directories where libraries can be located.

Currently, we have only 3 locations where LDF looks for libs https://github.com/platformio/platformio/blob/develop/platformio/builder/main.py#L91

Summary:

The order/priority:

  1. Project lib directory
  2. lib_extra_dirs
  3. Global lib
  4. Framework libs
glelouet commented 8 years ago

I'm not very good at python.

What I see from ( https://github.com/platformio/platformio/blob/develop/platformio/util.py ) is that

getprojectlib_dir should have its own resolution instead of static lib/, eg

def get_projectlib_dir():
    return _get_projconf_option_dir(
        "inc_dir",
        join(get_home_dir(), "lib")
    )

then this method could return a list of directories ( change the get_projconf_option_dirS to split the option_dir by ',' then map using expanduser) which would be appended the two other lib directories in main.py

I'm not very good at python so I don't try to do that right now, but I may give it a try if you can't afford.

ivankravets commented 8 years ago

Please don't look into the code. Please read my comment https://github.com/platformio/platformio/issues/537#issuecomment-188877749 and give me an answer about this approach in PlatformIO 3.0. I'll not change LDF in PlatformIO 2.0.

glelouet commented 8 years ago

Sorry I didn't get a question in previous comment, so to be constructive I looked at your link which actually was some code. It took me some time and I feel awkward when you tell me to look at code then to not to after I did look at code.

Please rephrase your question. I can rephrase my issue if you don't get my point, as it seems we don't understand each other well.

My previous comment proposed to modify code in utils.py. It was composed of two successive modifications, the second one being optionnal

Those modifications do not require to modify LDF, furthermore they only ADD functions (the inc_dir variable is optionnal) and thus do not require a change in the main version number.

ivankravets commented 8 years ago

@glelouet I've just tried to re-read your https://github.com/platformio/platformio/issues/537#issue-136387614.

possible solution is to have two environment variables : lib_dir to specify where we PUT the required libs

This is equal to the default value /.platformio/lib

and inc_dir to specify where are our private library not managed by platformio downloader.

This is equal to the lib folder from the project

This way, I could set inc_dir to ../lib (or ../lib, ./lib, ../../lib with a list) in my platformio.ini , and set the lib_dir in the cli if modification required (eg using arduino IDE)

If you don't want to put any info into platformio.ini, you can make symbolic links to that folders.

glelouet commented 8 years ago

If you don't want to put any info into platformio.ini, you can make symbolic links to that folders.

You don't understand me, and I'm tired of trying to explain myself. If you want to discuss, read my whole issue again. You don't get me at all, I said I DO want to change the plaformio.ini

ivankravets commented 8 years ago

Please provide archive with example project

glelouet commented 8 years ago

why ?

ivankravets commented 8 years ago

Dear @glelouet,

  1. If you open issue and know that "PlatformIO member" You don't understand me - you have to explain in details, otherwise issue will not be resolved.
  2. I've got you since the 1-st your comment. What is more, I explained you twice why it doesn't work in PlatformIO 2.0 and how could work in PlatformIO 3.0. However, you don't want listen me and answer in each comment "need to implement inc_dir for private libs".

Let me rephrase:

  1. PlatformIO 2.0 has PUBLIC & PRIVATE libs support.
  2. The public libs are located in PLATFORMIO_HOME/lib by default. However, user have option to change this path in platformio.ini via lib_dir.
  3. The PRIVATE libs are located in the project's lib directory and you can't change it. What is more, this lib contains readme.txt with DETAILED explanation how to use it, like ...This directory is intended for the project specific (private) libraries....

As I've said above, I don't like current PlatformIO 2.0 scheme with these lib folders. In this case, I propose these options:

_And the last... _

You want to have "OWN" PRIVATE INC dirs. No problems. See my https://github.com/platformio/platformio/issues/537#issuecomment-188877749. According to it you will have ability to specify OWN EXTRA LIB dirs. You can call them "PRIVATE" or even "SPONGE BOB DIRS". For example,

[env:myenv]
platform = ...
board = ...
lib_extra_dirs = /path/to/private/dir1,/path/to/private/dir2,/path/to/sponge/bob/dir

Now, please forget about "i want inc_dir" and try to listen what I propose. If you have corrections/comments to my proposition, I'm glad to correct it.

P.S: Guys, @FWeinb, @marvinroger, @dave-newson, @furyfire, @oskargargas, @sarfata, @jasonmhite what do you think?

marvinroger commented 8 years ago

You don't understand me, and I'm tired of trying to explain myself.

@glelouet I am pretty sure working on such an important _open-source_ project and getting this kind of answers is tiring too. Just saying. Anyway, GitHub is one of these few rare places on the Web where constructive discussions are possible, so let's not bicker.

@ivankravets what you propose sounds good and reflected to me :+1:

dave-newson commented 8 years ago

@glelouet I'm having trouble reading the original request. If I'm reading you right, you've got two concerns;

Going back to your original requirements (I've fiddled with the names because the terminology is confusing);

My understanding is that the Arduino IDE is an ignorant beast, and it'll only include libs which sit inside ~/.arduino/libs.

lib_dir to specify where we PUT the required libs, and inc_dir to specify where are our private library not managed by platformio downloader. This way, I could set inc_dir to ../lib (or ../lib, ./lib, ../../lib with a list) in my platformio.ini , and set the lib_dir in the cli if modification required (eg using arduino IDE)

I'm not understanding how this would solve your problem. Using this method it sounds like you would be telling PlatformIO to use/place libs from/in ~/.arduino/libs, but the inverse - being able to compile using Arduino IDE - is still not possible, as it won't resolve your paths of ../lib/whatever.

Concerning Arduino IDE compatibility, even if you tell platformio to dump libraries in the ~/arduino/libs folder, Arduino IDE still can't see your ../shared_libs because it doesn't know to look for them.

You could work around this issue by using symlinks (yay for linux) or in PlatformIO 3 you can fold ../shared_libs into its own project/repo, and use PlatformIO to semver manage that dependency. That way your ../shared_libs becomes ./libs/shared_libs, and I'd say this is the right way (although not the quick way) to do things. Otherwise your dependency is not managed at all.

@ivankravets suggestion of using a lib_extra_dirs config will partially solve your ./libs vs ./private_libs problem. In fact you can use this solution for your ../shared_libs too, but you'll still be screwed when it comes to Arduino IDE.

I would expect the following to work just fine:

[platformio]
lib_extra_dirs = ./private_libs,../libs

Honestly, I can't see a way to fix compatibility with the Arduino IDE if you keep using the ../libs (../shared_libs) directory. You'd be safer making this its own lib, and then a direct dependency of your project. That's not currently possible in PlatformIO2, but should show up in PlatformIO3.

Bear in mind I'm not very familiar with the Arduino IDE because I find it horrible to use. If you know something we don't about how to include libraries, let us know.

glelouet commented 8 years ago

@dave-newson

My understanding is that the Arduino IDE is an ignorant beast, and it'll only include libs which sit inside ~/.arduino/libs.

Well, it also has a list of libraries folder to look into.

but the inverse - being able to compile using Arduino IDE - is still not possible, as it won't resolve your paths of ../lib/whatever.

following this blog ( http://www.ikravets.com/computer-life/platformio/2014/10/07/integration-of-platformio-library-manager-to-arduino-and-energia-ides ) it would. (short : setting the lib_dir to ~/Arduino/Libraries )

@ivankravets

You want to have "OWN" PRIVATE INC dirs. No problems. See my #537 (comment). According to it you will have ability to specify OWN EXTRA LIB dirs. You can call them "PRIVATE" or even "SPONGE BOB DIRS".

I didn't understand the [env:xxx] and it was about platformio 3 . My issue is about platformio 2, I looked into the code to help you find a solution. If you say the behaviour changes, then I need to understand this new behaviour, but I don't and this is not my issue.

I only had ONE issue with platformio 2 , and this was, that I could not override the ./lib/ as being the SPONGE BOB folder. Maybe the example of code I gave you could help, maybe it was stupid ; but in the end you throw it away and this is rude.

@marvinroger

where constructive discussions are possible, so let's not bicker.

Yes but when I feel like my point is out of the discussion there is no reason for me to try to discuss. I don't want to waste time, nor to waste yours - platformio is good and I'm sure you can enhance it.

I unsuscribe, just realized I didn't get nor bring anything good from this.

ivankravets commented 8 years ago

@glelouet Why did you close this issue? Please read my comments above. I don't have any objections to resolve this issue but it will be done in PlatformIO 3.0. Also, please read carefully https://github.com/platformio/platformio/issues/537#issuecomment-189708570 and I wait for the answer: "Will PlatformIO 3.0 resolve your issue?" according https://github.com/platformio/platformio/issues/537#issuecomment-189708570

P.S: You are right, let's save time each other. What is more, all PlatformIO users are equal for me. I mentioned a few PlatformIO users above "not for the help to humble @glelouet", but for the constructive answer as for https://github.com/platformio/platformio/issues/537#issuecomment-189708570.

I don't know these users, but they are authors/commentors of a few issues related to PlatformIO 3.0 Library Manager. See https://github.com/platformio/platformio/labels/lib

As you can see, I don't have aim to "discard all users requests" but quite the contrary I'm grateful them for the comments, ideas and corrections. The only this friendly and politely discussions in our community will help PlatformIO Team to create PlatformIO FOR THE ALL, but not only for @ivankravets.

Regards, Ivan.

dave-newson commented 8 years ago

@glelouet don't be discouraged, your requests are still valid. The biggest problem with this issue is the terminology and requirements are slightly confusing, so unfortunately you will need to bear with us while we confirm exactly what the issue is, and if it's something PlatformIO can/should resolve.
Sometimes it's necessary to talk slow using simple words and bullet points. Write a story, paint us a picture. If understanding people was easy we wouldn't have wars.

following this blog ( http://www.ikravets.com/computer-life/platformio/2014/10/07/integration-of-platformio-library-manager-to-arduino-and-energia-ides ) it would. (short : setting the lib_dir to ~/Arduino/Libraries )

Thanks for the link, that's very informative about what it is you're trying to accomplish.

Unfortunately, I think this still doesn't address the problem of Arduino IDE not being able to see the ../shared_libs directory, so my point still stands. You would instead need to make your ../shared_libs into its own project, eg. glel_shared_code and then include that as a dependency in your PlatformIO project. Doing that would ensure it is added to ~/arduino/libs by PlatformIO, and then Arduino IDE would see it.

With that covered, let's ignore the issue of Arduino IDE's terrible library management and address this whole lib_extra_dirs thing. With the libs_extra_dirs you would be able to tell PlatformIO to include libraries from a number of additional locations. This would let you have your ../shared_libs, ./libs and ./private_libs folders, while PlatformIO only installs 3rd party libs to ./libs. (Unfortunately even with that feature Arduino IDE is still trash. You still need all your libs to live in ~/arduino/libs for Arduino IDE to see it)

I only had ONE issue with platformio 2 , and this was, that I could not override the ./lib/ as being the SPONGE BOB folder. Maybe the example of code I gave you could help, maybe it was stupid ; but in the end you throw it away and this is rude.

I think Ivan is really trying to concentrate his efforts on bringing PlatformIO3 into production, as it'll fix a massive range of issues rather than just this one concern, so he's looking for how a feature could be added to PlatformIO3 which will solve this issue, rather than continuing to work on deprecated PlatformIO2 code, which will only be thrown away in the coming months.

Unfortunately with OSS, unless you're willing to make your own contributions, it can mean waiting until the authors goals align with yours and also coming to a compromise with them.

As an aside, using CLion I achieved the same ability - adding multiple lib dirs in PlatformIO2, by adding the following configs:

platformio.ini

[env:xxx]
build_flags = -L../libs

and I think I also used: cmakelists.txt

add_subdirectory('../libs')

Well, it also has a list of libraries folder to look into.

How do you set this? I Googled for it and I couldn't find anything. Can you provide some details? Is it an env var or something?

\/\/\/\/\/ Thanks for the clarification.

ivankravets commented 8 years ago

@dave-newson

build_flags = -L../libs

That is compiler/preprocessor flag. PlatformIO Library Dependency Finder doesn't handle it. We have 2 different meaning under lib in PlatformIO:

  1. lib is compiler compatible library, like mylib.a, mylib.so and etc. -L is the path where compiler will look for *.a libs. What is more, you can specify which libs should be picked up via -lmylib. BUT.... This is really advanced usage and I don't recommend to use it if you don't understand what doest is mean.
  2. lib is raw source code of the future library. PlatformIO makes REAL lib.a from your source code automatically. Take a look at .pioenvs/*/.

As for Arduino IDE & PlatformIO. @glelouet is right, we can share the same folder between PlatformIO & Arduino IDE without problems.

The issue here that @glelouet wants to keep "some private" libs between both sides. I hope that PlatformIO 3.0 will help him to achieve it.

@glelouet please answer. Thanks.

glelouet commented 8 years ago

I gave up arduino IDE and gave Atom+Platformio a try. It's ok so far, still missing the features of eclipse(autocomplete is …strange) but I'm fine with it as of now.

@ivankravets

platformio.ini > [platformio] > lib_dir will relate to project/lib directory, not to global as it is.

well I wont discuss the variable name, I think separating global and project-specific libs like this is good

GLOBAL libs folder, just change platformio.ini > [platformio] > home_dir, where GLOBAL dir will equal to home_dir/lib.

Well, I think we should have a separation between platformio home_dir and the corresponding global_dir. What happens if I download one lib for different architectures ? Do they share the same lib source ? Is there a way to expand variables in platformio.ini, like global_dir=$(home_dir)/arduino/libs ?

Besides those may-be issues, it seems good to me.

ivankravets commented 8 years ago

Temporary link to development version of PlatformIO 3.0 docs http://docs.platformio.org/en/feature-platformio-30/projectconf.html#lib-extra-dirs

mohsh86 commented 8 years ago

this thread got me confused as hell, i have a couple of questions:

  1. is lib_extra_dirs implemented yet in PlatformIO 2.11 ?
  2. is pio lib install -g/--global is not yet implemented in PlatformIO ? i ran it on lib 89 (Pub sub) and it got installed in ~/.platformio/lib , is that the default ?
  3. i have my own library that i am sharing across different project, is there any current way on current version (2) to include them into project without copying or creating symbolic link to the project lib folder ?

note: build_flags = -L

didn't work as well

ivankravets commented 8 years ago

@mohsh86 PIO3 pio lib install -g/--global command is equal to pio lib install command in PIO2.

You can place temporary your own libraries to ~/.platformio/lib and after PIO3 release you can create extra library storage and specify it in lib_extra_dirs.

PIO3 is planned for release tomorrow.

mohsh86 commented 8 years ago

awesome dude, you rock !