building using --from-cache is not working: "Persistent cache missing file" error

[mickours]

When I try to build my recipe from a previously generated cache (using --from-cache option) I get the following error message: Persistent cache missing file

In the code it says: # This error can be due to the improper format of the file cache_cmd_index

And in fact, cache_cmd_index only contains:

--- []

[mickours]

You use to mark it as "ready", did you look at this?

I just tried again and I get the same error. Here is the detailed logs:

...
[debug] Starting proxy cache server for macrostep 'kameleon_customization'...
[debug] Creating  cache directory kameleon_customization for Polipo
[debug] Starting web proxy Polipo in directory /home/mercierm/Projects/test-kameleon/build/cache/DATA/kameleon_customization using port: 8001
[debug] Starting process '["/usr/bin/polipo", "-c", "/dev/null", "diskCacheRoot=/home/mercierm/Projects/test-kameleon/build/cache/DATA/kameleon_customization", "maxDiskCacheEntrySize=-1", "idleTime=1", "allowedC
lients=0.0.0.0/0", "proxyPort=8001", "daemonise=false", "proxyAddress=0.0.0.0", "logFile=/home/mercierm/Projects/test-kameleon/build/polipo-root.log", "logLevel=0xFF", "proxyOffline=false"]'
[debug] The proxy is not responding. Server response: Failed to open TCP connection to 127.0.0.1:8001 (Connection refused - connect(2) for "127.0.0.1" port 8001)
[debug] The proxy is responding
[debug] The proxy is responding
Step 21 : setup/kameleon_customization/configure_bashrc
--> Running the step...
[in] ++ dirname /etc/skel/.bashrc
[in] + mkdir -p /etc/skel
[in] + save_env
[in] + set +x
[info] Exit status : 0
Redirecting pipe into cache
Persistent cache missing file
fatal error...
Waiting for cleanup before exiting...
Cleaning setup section
...

The /home/mercierm/Projects/test-kameleon/build/cache/DATA/kameleon_customization directory actually contains a lot of ascii and Python files (60 files!):

ls  | sudo xargs file                                                                                                                                 
pipe-cat_home_mercierm_pro20151127-25801-1a2cf8l: ASCII text
pipe-cat_home_mercierm_pro20151127-25801-1j5npl2: Python script, ASCII text executable
pipe-cat_home_mercierm_pro20151127-25801-1nkp5qc: ASCII text
pipe-cat_home_mercierm_pro20151127-25801-3dcvrs:  Python script, ASCII text executable
pipe-cat_home_mercierm_pro20151127-25801-4e8mug:  ASCII text
pipe-cat_home_mercierm_pro20151127-25801-iif9a7:  ASCII text
pipe-cat_home_mercierm_pro20151130-13664-10s3sc0: ASCII text
pipe-cat_home_mercierm_pro20151130-13664-1hzkmee: ASCII text
pipe-cat_home_mercierm_pro20151130-13664-1x1zndt: Python script, ASCII text executable
pipe-cat_home_mercierm_pro20151130-13664-1y0690t: Python script, ASCII text executable
...

Something is definitely messy with the cache implementation: It has kept all the file in the cache since last November even if it do not changes AND it puts it on the exported cache. Even so, it do not work...

We should take some time to reimplement this part. Maybe after the g5k school.

[mickours]

Ok I just retried with a clean cache (not polluted with previous run) and it works. The cache_cmd_index contains data so it looks like it is the root problem. Here is the working index:

cat build/cache/metadata/cache_cmd_index 
---
- :raw_cmd_id: 1568dcdb3805949eeb9faac19d672e5b8892726f
  :stdout_filename: pipe-cat_home_mercierm_pro20160108-20042-kjoz8l
- :raw_cmd_id: 1141ce3880556eb39e89c599c28a2d879e94c8f3
  :stdout_filename: pipe-cat_home_mercierm_pro20160108-20042-15m9yja
- :raw_cmd_id: 0ef7d99601bfe5f2ffb9d122b15d83e99e737d89
  :stdout_filename: pipe-cat_home_mercierm_pro20160108-20042-1scw4d1
- :raw_cmd_id: 0bfb1cd3b7d72fb05ba8af5fbfa46ec4d5d32c9a
  :stdout_filename: pipe-cat_home_mercierm_pro20160108-20042-1i6eiwj
- :raw_cmd_id: c3aa2ef5ffc15c746c355de44e6b896362bba8b5
  :stdout_filename: pipe-cat_home_mercierm_pro20160108-20042-ts7zpz
- :raw_cmd_id: db7ab36b1f93d64a1426309dfa6233e009402f45
  :stdout_filename: pipe-cat_home_mercierm_pro20160108-20042-1y74c5d
- :raw_cmd_id: c5f51fcea78ca3d498d454996ed0c4a91aa866f1
  :stdout_filename: pipe-cat_.vimrc20160108-20042-jiq82l
- :raw_cmd_id: 0b8455a89a866f4812d4e1733ee5c8b5d3d02b50
  :stdout_filename: pipe-cd_.vim_bundle_tar_cf20160108-20042-mlfl5b
- :raw_cmd_id: 81158e8c7dd4d4edc29332a32d1e57ec720f5946
  :stdout_filename: pipe-cat_.tmux.conf20160108-20042-bbfeeg
- :raw_cmd_id: 3490accd123750e9214aff92b2c428201a661ec9
  :stdout_filename: pipe-cat_.gitconfig20160108-20042-3bm95n

[SalemHarrache]

I haven't started developping yet, the "Ready" status displays because I work with https://waffle.io/ to organize my tasks.

For your Python files, the DATA folder only contains the files copied via ''pipe''. There are 60 files because several builds are behind this archive. It's an issue we're having since the beginning.

We have two solutions here : either we make sure to have one cache folder by recipe (but we don't take advantage of the cache between recipes, i.e debootstrap, etc.) or we pool this cache folder, but then the archive becomes bigger for nothing (since it contains only one recipe).

It's not easy to make a decision on this, but for now a good workaround can be used as an alternative that's what I always do) and once my recipe is over, I make a new build from scratch to create my cache archive.

We should take some time to reimplement this part. Maybe after the g5k school.

+1