[question] Source code for Air Gapped system

[arturngomes]

What is your question?

Hi, I'm trying to setup conan and artifactory in an air gapped system. So far I managed to get all the dependencies that I need in the self hosted repository. However when I try to install a project that requires some of the dependencies I run into the issue of not being able to compile the project, because the source code is in github, and being in an air gapped system means I have no way of accessing it from the that machine.

Example:

1. Setup a project that has grpc/1.65.0 which depends on abseil/20240116.2 (Assuming all the packages have been correctly imported into the air gapped system)
2. Run conan install . --output-folder=build --build=missing
3. Error: abseil/20240116.2: WARN: network: Error downloading file https://github.com/abseil/abseil-cpp/archive/20240116.2.tar.gz: 'HTTPSConnectionPool(host='github.com', port=443): Max retries exceeded with url: /abseil/abseil-cpp/archive/20240116.2.tar.gz (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f00674c1cd0>: Failed to establish a new connection: [Errno -2] Name or service not known'))'

Is it possible to pre-download these sources and upload as part of the conan cache save package archive to the self hosted repository?

Have you read the CONTRIBUTING guide?

• [X] I've read the CONTRIBUTING guide

[memsharded]

Hi @arturngomes

Thanks for your question.

Have you checked the "sources backup" feature? I think it can help with your air-gap system, it was designed with that use case in mind too, depending on how you are "importing" the packages (from ConanCenter I guess), it might be quite straightforward.

Please have a look at https://docs.conan.io/2/devops/backup_sources/sources_backup.html and let us know.

[arturngomes]

Hi @memsharded,

Thanks a lot for the quick reply. After reading through the link you sent this is the process I'm following (example for the abseil library):

Downloading packages:

1. Add tools.build:download_source=True to global.conf
2. conan graph info --requires=abseil/20240116.2 # I noticed this will actually download the sources and package in one go
3. conan cache save abseil/20240116.2 --file abseil_20240116.2.tgz

Question: Is it possible use the conan download in a way that forces to download the sources?

Then I move the abseil_20240116.2.tgz into the air gapped system and upload it there into the self hosted artifactory.

Uploading packages:

1. conan cache restore abseil_20240116.2.tgz
2. conan upload abseil/20240116.2 -r local

I guess I'm still missing something since when I upload on the air gapped system I get: -------- Uploading backup sources -------- No backup sources files to upload

edit: I can see the sources are actually in the file:

[memsharded]

Add tools.build:download_source=True to global.conf

Probably not recommended, it is inefficient to always download the sources in every conan install. You can add this argument in command line like -c:a tools.build:download_source=True just when you want.

conan cache save abseil/20240116.2 --file abseil_20240116.2.tgz

Then I move the abseil_20240116.2.tgz into the air gapped system and upload it there into the self hosted artifactory.

I see. The conan cache save doesn't save the download cache. At the moment it is necessary to:

• Define the core.sources:upload_url
• And upload the packages.

I think you might try this way:

• Define a known location of the backup sources cache defining core.sources:download_cache
• Together with the conan cache save packages, copy to that folder to the air gap
• Make sure to define core.sources:download_cache in the air-gap pointing to the copied folder
• Define core.sources:upload_url to the generic repo in the airgap
• Now the conan upload should upload the sources too

Please try that and let us know.

[arturngomes]

Hi @memsharded,,

Once again thank you for the quick reply. I finally managed to get it to work thanks to your help.

Just in case someone else has the same issue, I'll leave here the procedure I followed to be able to get it working.

Assumptions:

• Air gapped system is setup and repositories (using artifactory CE) have been setup
• conan login to the repository has been used
• source_credentials.json is setup (for uploading the backed up source code)

Replace the variables below identified by with whatever you need.	variable	description
cache_location	Any directory where the source code (from github/etc) will be stored
package	The name of the package
version	The version of the package
out_file	The name of the compressed file where to store the cached value (e.g abseil_20240116.2.tgz)
packaged_zip	Any name you want to identify the packages you just downloaded (e.g abseil_20240116.2.zip)
upload_tmp_dir	Directory used to unzip the packaged_zip before uploading
backup_url	The url of the generic repository for artifactory
remote_name	The name of the remote repository to be used

Downloading Packages

For this part a machine with internet connection is needed.

1. Download the packages needed and their source code, probably also fine to just run the conan install with a conanfile.txt

   conan graph info --requires=${package}/${version} \
   -c:a tools.build:download_source=True \
   -cc core.sources:download_cache=${cache_location}

2. Save the package to a tgz file:

   conan cache save ${package}/${version} --file ${out_file}

3. Zip everything:

   zip -r ${packaged_zip} ${out_file} ${cache_location}

4. Copy this zip file into the air gapped system

Uploading Packages

First go to a directory where you can extract the package and copy the zip file into it:

1. Extract the zip file locally:

   unzip ${packaged_zip}

2. Restore the package into the cache before uploading:

   conan cache restore ${out_file}`

3. Upload the package to the air gapped repository:

   conan upload ${package}/${version} -r ${remote_name}\
       -cc core.sources:download_cache="${upload_tmp_dir}/${cache_location}" \
       -cc core.sources:upload_url="${backup_url}"/ \
       -cc 'core.sources:download_urls=["${backup_url}", "origin"]'

Using the Packages

1. run the install that requires the packages:

   conan install . --output-folder=build --build=missing -cc 'core.sources:download_urls=["${backup_url}", "origin"]'

[memsharded]

Many thanks for your feedback, this is great.

I see some potential simplifications. For example, it wouldn't be necessary to do operations one package at a time. A "package list" can represent multiple packages and conan cache save/restore can use "package lists" as argument to do their operation, as well as conan upload and other commands also can use package lists

I also see some potential improvements and features in Conan, like the possibility of conan cache save the download cache too. However, this doesn't seem a high priority at the moment, as there is a viable process.

In any case, I'd like to comment on something quite important. If you are using an air-gapped environment I am sure that you have high security requirements and other policies and constraints. I am not sure, but if the packages that you are bringing into the air-gap are packages from ConanCenter, then most likely, the recommended approach for your case would be different, mostly building the packages from source yourself. We have different docs and features in Conan to help with this, please check:

• https://docs.conan.io/2/devops/using_conancenter.html
• https://docs.conan.io/2/devops/conancenter/hosting_binaries.html
• https://docs.conan.io/2/devops/devops_local_recipes_index.html