greiginsydney / Intervalometerator

A multi-purpose time-lapse DSLR camera controller
GNU General Public License v3.0
54 stars 9 forks source link

Images not transferring from camera -> Pi when using a 4G wifi dongle #79

Closed greiginsydney closed 3 years ago

greiginsydney commented 3 years ago

I am also trying to get things working over a 4g wifi dongle and over traditional wifi but without port forwarding. I removed the fixed ip settings on the pi and installed remote.it on the pi. I could then set up a connection for ssh, http on port 80 and https on port 443. everything appeared to be working fine as I was able to access the camera via remote.it and change settings etc. However I then noticed that no photo's were getting transferred to dropbox. When I checked the photos have stayed on the camera and not gone to the pi. attached are the cameraTransfer.log and the gunicorn.error files. Any help appreaciated. cameraTransfer.log.txt gunicorn.error.txt

Originally posted by @aerialvideographyuk in https://github.com/greiginsydney/Intervalometerator/issues/8#issuecomment-815252097

greiginsydney commented 3 years ago

Hi Dave,

Can you please attach the file from /var/log/celery/celery_worker.log, as that's where the camera tranfer process is logged in detail? (If the file's large, please remove everything prior to ~20:15 on the 7th). Alternatively, look for Received task: intvlm8r.copyNow and only send the rest from that point.

You mentioned to me elsewhere that changing IPs and unstalling/uninstalling remote.it caused then resolved the issue. Can you point me to the doco for that, as I'm keen to understand what changed and why it broke.

- G.

greiginsydney commented 3 years ago

BTW, the two files you've already included show the process kicked off OK:

  1. cameraTransfer.py fired, and raised a request with intvlm8r.py for the transfer, the latter responding in the affirmative ('OK').
  2. intvlm8r.py saw that and fired off a background task for a 'celery' worker to handle the transfer.
  3. The bouncing ball leads us to what happened next ...
aerialvideographyuk commented 3 years ago

Hi Greig The celery_worker.log hadn't been updated since 5th April, the day I put remote.it on. I removed the remote.it package and rebooted the pi and everything is working again as far as I can tell. Will leave it run today and then see what I'm doing wrong with remote.it.

aerialvideographyuk commented 3 years ago

dont know if it's connected but if I go to the transfer tab on the web interface and then do a copy now I get the background task pending etc messages then image

greiginsydney commented 3 years ago

Well, that's definitely a bug. I fixed one of those recently in v4.0.2 ("Fixed error in copyNow when camera has no images"). Are you running that version or one earlier? The celery log will ID the cause, when you can get one.

greiginsydney commented 3 years ago

@aerialvideographyuk :

Will leave it run today and then see what I'm doing wrong with remote.it.

It's more likely an incompatibility between the two, but it would be good to understand what's going on and try to get them to place nicely together. Can you link me to the installation process/guide you followed?

aerialvideographyuk commented 3 years ago

https://support.remote.it/hc/en-us/articles/360047542051-Installing-remoteit-on-a-Raspberry-Pi-running-Raspbian-or-Raspberry-Pi-OS Is what I used - reinstalling now - will let you know how it goes

aerialvideographyuk commented 3 years ago

I have reinstalled and again registered ssh, http, and https and it all appears to be working fine. Will leave it run through a complete day and see if it's OK. As far as I'm aware I didn't do anything different to the first time. I'm running version 4.01 by the way. I will update to 4.0.2 once I see that all is OK with remote.it.

aerialvideographyuk commented 3 years ago

I spoke too soon. Once again it looks like the copy from camera to pi has failed to take place. Will check the logs and update as soon as I can...... removed remote.it again and all now working. There seems to be some incompatibility so not sure how the other chap got it working.

greiginsydney commented 3 years ago

You didn't enable http-alt by any chance?

The intvlm8r uses port 8080 for the transfer process... (It's easily-enough moved if that's the issue.)

image

aerialvideographyuk commented 3 years ago

No I didn't. Was looking at the camera transfer code last night and was wondering if I needed to enable a connection to 8080 on remoteit? Have you managed to get it working?

greiginsydney commented 3 years ago

Yes, I was just able to login to the intvlm8r remotely (VERY neat!!) and issue a 'CopyNow', which then copied the images and created the thumbs.

I'll set the daily copy now time for the next hour and see if that takes place.

(Have you upgraded to 4.0.2 yet??)

aerialvideographyuk commented 3 years ago

No I haven't upgraded yet but I did put remoteit back on with port 8080, port 22 and port 80 connections. I could also do a copy and a transfer took place after a reboot. Will leave it run a cycle again and see if it works. What is port 443 used for? Do I need it?

greiginsydney commented 3 years ago

443 is used for if you were using SSL (a certificate for "https").

You'll definitely break the intvlm8r if you enable 8080 for remote.it, as the scheduled cameraTransfer.py script calls intvlm8r.py on port 8080.

This will tell you if anything other than the intvlm8r is using port 8080:

pi@BenchPi3Bplus:~ $ sudo netstat -pant | grep 8080
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      548/nginx: master p

So that's telling us that process 548 - nginx - is listening on 8080.

For more details on process 548 (this number will change):

pi@BenchPi3Bplus:~ $ ps aux | grep 548
root       548  0.0  0.2  50232  2268 ?        Ss   Apr11   0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
pi         673  0.2  3.7  60356 35548 ?        S    Apr11   7:13 /usr/bin/python3 -m celery -A intvlm8r.celery worker --logfile=/var/log/celery/celery_worker.log --loglevel=DEBUG --time-limit=300 --concurrency=8 -n celery_worker@BenchPi3Bplus --pidfile=/var/run/celery/celery_worker.pid --executable=/usr/bin/python3
pi        7064  0.0  0.0   7348   508 pts/0    S+   15:24   0:00 grep --color=auto 548
pi@BenchPi3Bplus:~ $

(Only the top line is relevant - the other two are partial matches for '548'. This use of 8080 is declared in the file /etc/nginx/sites-available/intvlm8r.)

If you can send me some screen-scrapes (like those above) showing remote.it is using 8080, I'll wrap that all up in a new FAQ and document how to change the ports in the intvlm8r.

- G.

aerialvideographyuk commented 3 years ago

I did a complete re-install of the latest code - all worked fine. (I have not put SSL on though) I then put remoteit back on and again the same results - no copy. Did as you asked above and only one process running but appears to be on python2 and no celery process which ties up with the celery log not updating.

pi@AVUKcontroller1:~ $ sudo netstat -pant | grep 8080
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      502/nginx: master p
pi@AVUKcontroller1:~ $ ps aux | grep 502
root       480  0.7  3.4  22032 15024 ?        Ss   21:29   0:03 /usr/bin/python2 /usr/bin/supervisord -n -c /etc/supervisor/supervisord.conf
root       502  0.0  0.4  50216  2188 ?        Ss   21:29   0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
pi         743  0.0  0.4   7332  2024 pts/0    S+   21:38   0:00 grep --color=auto 502
pi@AVUKcontroller1:~ $
greiginsydney commented 3 years ago

That's weird. But if you back up a step, without remote.it and the scheduled transfers working fine, what do netstat & ps aux reveal?

PLEASE update the intvlm8r though. I don't want either of us to waste any more time on something that could have already been resolved (explaining why I'm not seeing it here).

[BTW, any resolution might be in an underlying component, not my own code. There are a lot of moving pieces here and my code is a tiny portion of that. A sudo apt-get update && sudo apt-get upgrade -y could be your fix, as I'm developing and testing with Raspbian & all the other components at their current releases].

greiginsydney commented 3 years ago

Sorry Dave, scratch that update request bit. I hadn't read your "complete reinstall" comment correctly.

aerialvideographyuk commented 3 years ago

OK removed remote.it again and run the 2 commands when it was copying files and when it finished

When still copying files pi@AVUKcontroller1:~ $ sudo netstat -pant | grep 8080 tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 450/nginx: master p tcp 0 0 127.0.0.1:8080 127.0.0.1:36594 TIME_WAIT -

pi@AVUKcontroller1:~ $ ps aux | grep 450 root 450 0.0 0.2 50216 900 ? Ss 18:37 0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on; pi 634 0.0 0.4 7332 2036 pts/0 S+ 18:43 0:00 grep --color=auto 450 pi@AVUKcontroller1:~ $

Once copy finished pi@AVUKcontroller1:~ $ sudo netstat -pant | grep 8080 tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN 450/nginx: master p

pi@AVUKcontroller1:~ $ ps aux | grep 450 root 450 0.0 0.0 50216 316 ? Ss 18:37 0:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on; pi 642 0.0 0.4 7332 1904 pts/0 S+ 18:47 0:00 grep --color=auto 450 pi@AVUKcontroller1:~ $

As far as I can tell I'm using the same as you except I don't have an SSL certificate. Not sure where to go next now.

greiginsydney commented 3 years ago

I'm lost too. Mine's behaving fine. Here's the start of the celery_worker.log at 3pm, when my daily transfer is scheduled to commence:

[2021-04-17 15:00:02,009: DEBUG/MainProcess] pidbox received method active() [reply_to:{'exchange': 'reply.celery.pidbox', 'routing_key': 'a1a8db10-2391-30ca-b4be-ca565a852084'} ticket:fc97eb54-e07e-466a-b3d9-cc69137ecc47]
[2021-04-17 15:00:02,035: INFO/MainProcess] Received task: intvlm8r.copyNow[e0ed5ce0-c3fa-46f1-af05-d775957be1de]  
[2021-04-17 15:00:02,036: DEBUG/MainProcess] TaskPool: Apply <function _trace_task_ret at 0x75666a98> (args:('intvlm8r.copyNow', 'e0ed5ce0-c3fa-46f1-af05-d775957be1de', {'lang': 'py', 'task': 'intvlm8r.copyNow', 'id': 'e0ed5ce0-c3fa-46f1-af05-d775957be1de', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [1800, None], 'root_id': 'e0ed5ce0-c3fa-46f1-af05-d775957be1de', 'parent_id': None, 'argsrepr': '()', 'kwargsrepr': '{}', 'origin': 'gen753@BenchPi3Bplus', 'reply_to': '8c03f697-bae8-3041-9fdf-be80316ca4b1', 'correlation_id': 'e0ed5ce0-c3fa-46f1-af05-d775957be1de', 'hostname': 'celery_worker@BenchPi3Bplus', 'delivery_info': {'exchange': '', 'routing_key': 'celery', 'priority': 0, 'redelivered': None}, 'args': [], 'kwargs': {}}, b'[[], {}, {"callbacks": null, "errbacks": null, "chain": [{"task": "intvlm8r.newThumbs", "args": [], "kwargs": {}, "options": {"task_id": "427aa427-6026-4fe2-b7a8-bbc275bc1415", "reply_to": "8c03f697-bae8-3041-9fdf-be80316ca4b1"}, "subtask_type": null, "chord_size": null, "immutable": true}], "chord": null}]', 'application/json',... kwargs:{})
[2021-04-17 15:00:02,041: DEBUG/MainProcess] Task accepted: intvlm8r.copyNow[e0ed5ce0-c3fa-46f1-af05-d775957be1de] pid:648
[2021-04-17 15:00:02,544: INFO/ForkPoolWorker-7] copyNow(): entered
[2021-04-17 15:00:02,545: DEBUG/ForkPoolWorker-7] (gp_port_new) Creating new device...
[2021-04-17 15:00:03,547: INFO/ForkPoolWorker-7] copyNow() trying to init the camera

(I'm logging at the very verbose debug level, which is why it might look a little different to yours).

When you install and configure remote.it, are you doing anything in the Pi's config other than sudo apt install remoteit that I've not realised? (I'm only installing it, browsing to the pi on port 29999 and then setting up remote.it from the browser).

Maybe @cadpix can share the secret of their deployment?

- G.

greiginsydney commented 3 years ago

In the meantime, let's see if we can narrow down where this is falling off the rails.

Can you please edit www/intvlm8r.py to add the logging lines to def copyNowCronJob() as shown here?

@app.route("/copyNow")
def copyNowCronJob():
    """
    This 'page' is only one of two called without the "@login_required" decorator. It's only called by 
    the cron job/script and will only execute if the calling IP is itself/localhost.
    """

    app.logger.debug('=== Yes, /CopyNow called')

    sourceIp = request.environ.get('HTTP_X_REAL_IP', request.remote_addr)
    if sourceIp != "127.0.0.1":
        app.logger.debug('=== Source IP is bad. Aborting')
        abort(403)

    app.logger.debug('=== Source IP is good. Firing the async tasks now.')

    tasks = [
        copyNow.si(),
        newThumbs.si()
    ]
    chain(*tasks).apply_async()

    res = make_response('OK')
    return res, 200

This will help me understand if the scheduled task is firing OK and the correct page is being called.

Update your Copy Hour on the /Transfer page to occur at the top of the next hour, come back then and see what (if anything) is logged in gunicorn.error.

Again, here's mine (and this is the immediate precursor to the celery log extract I posted in my previous comment):

[2021-04-13 15:00:01 +1000] [3947] [DEBUG] GET /copyNow
[2021-04-13 15:00:01 +1000] [3947] [DEBUG] === Yes, /CopyNow called
[2021-04-13 15:00:01 +1000] [3947] [DEBUG] === Source IP is good. Firing the async tasks now.

If you see the 'async tasks' message, that will lead me to look more closely at celery.

cadpix commented 3 years ago

Ours has been running continuously now for almost a year with only one site visit to replace a failed battery, however the transfer from the camera to Pi then Dropbox has been hit and miss. With that said it does eventually get there. I check it weekly and do need to hit the transfer now button a few times as left on its own it will only seem to transfer around 70 frames. Although it would be great to fix this I find this a small price to pay as I like to keep an eye on it using remoteit and once the transfer from the camera to the pi is done the pics make there way over to Dropbox. This is also much better than a two hour return trip to the camera in my case! My settings have the camera taking a pic every 15 mins, my suspicion is that once the pi starts the transfer, this is interrupted by the firing of the camera. It’s only a theory unfortunately!

aerialvideographyuk commented 3 years ago

I seem to be the odd one out :( I have attached some notes of what i did but below is the extract of gunicorn.error that you wanted.

[2021-04-18 23:06:32 +0100] [585] [DEBUG] Ignoring EPIPE [2021-04-18 23:07:38 +0100] [585] [DEBUG] GET /copyNow [2021-04-18 23:07:39 +0100] [585] [DEBUG] getCeleryTasks reports there are no activeTasks [2021-04-18 23:07:39 +0100] [585] [DEBUG] === Yes, /CopyNow called [2021-04-18 23:07:39 +0100] [585] [DEBUG] === Source IP is good. Firing the async tasks now.

The celery_worker.log has not been updated at all. If it helps I could leave the pi on and share the remote.it connection with you?

new camera.docx

greiginsydney commented 3 years ago

@cadpix:

I check it weekly and do need to hit the transfer now button a few times as left on its own it will only seem to transfer around 70 frames. Although it would be great to fix this I find this a small price to pay as I like to keep an eye on it using remoteit and once the transfer from the camera to the pi is done the pics make there way over to Dropbox. This is also much better than a two hour return trip to the camera in my case!

What version of intvl8mr are you running? Celery was added (in v4.0.0) to address this issue. I didn't realise it was still causing problems. If you're at 4.0.2 and it's still a problem, please raise a fresh issue and I'll look into it further.

My settings have the camera taking a pic every 15 mins, my suspicion is that once the pi starts the transfer, this is interrupted by the firing of the camera. It’s only a theory unfortunately!

Yes, I suspect the same. It's definitely better to schedule the copying of the images out of the camera for a period outside of the day's shooting schedule.

If you're not getting enough images out of the camera with each daily schedule, you might find creating a second manual entry will help. I've documented that on the FAQ page.

- G.

greiginsydney commented 3 years ago

@aerialvideographyuk:

I've just realised we've not checked that/if Celery is running.

Assuming your Pi still has remote.it installed, can you please execute sudo systemctl status celery and paste the output? (You'll need to hit "Q" to exit it).

I'm hoping for something that starts like this, with "active (running)" in soothing green. The absence of same gives us somewhere to go digging:

pi@BenchPi3Bplus:~ $ sudo systemctl status celery
● celery.service - Celery Service
   Loaded: loaded (/etc/systemd/system/celery.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2021-04-21 10:08:11 AEST; 4 days ago
aerialvideographyuk commented 3 years ago

Looks like it is failing to start.

pi@AVUKcontroller1:~ $ sudo systemctl status celery ● celery.service - Celery Service Loaded: loaded (/etc/systemd/system/celery.service; enabled; vendor preset: enabled) Active: activating (start) since Sun 2021-04-25 11:15:40 BST; 1min 19s ago Cntrl PID: 473 (sh) Tasks: 3 (limit: 877) CGroup: /system.slice/celery.service ├─473 /bin/sh -c /usr/local/bin/celery -A intvlm8r.celery multi start --logfile=/var/log/celery/%N.log --lo ├─475 /usr/bin/python3 /usr/local/bin/celery -A intvlm8r.celery multi start --logfile=/var/log/celery/%N.log └─587 /usr/bin/python3 -m celery -A intvlm8r.celery worker --detach --logfile=/var/log/celery/celery_worker.l Apr 25 11:15:40 AVUKcontroller1 systemd[1]: Starting Celery Service...

I hit q here but then re-ran the command to see if it started.

pi@AVUKcontroller1:~ $ sudo systemctl status celery ● celery.service - Celery Service Loaded: loaded (/etc/systemd/system/celery.service; enabled; vendor preset: enabled) Active: failed (Result: timeout) since Sun 2021-04-25 11:17:10 BST; 22h ago Process: 473 ExecStart=/bin/sh -c ${CELERY_BIN} -A ${CELERY_APP} multi start --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_NODES} ${CELERYD_OPTS} (code=killed, signal=TERM)

Apr 25 11:15:40 AVUKcontroller1 systemd[1]: Starting Celery Service... Apr 25 11:17:10 AVUKcontroller1 systemd[1]: celery.service: Start operation timed out. Terminating. Apr 25 11:17:10 AVUKcontroller1 systemd[1]: celery.service: Control process exited, code=killed, status=15/TERM Apr 25 11:17:10 AVUKcontroller1 systemd[1]: celery.service: Failed with result 'timeout'. Apr 25 11:17:10 AVUKcontroller1 systemd[1]: Failed to start Celery Service. pi@AVUKcontroller1:~ $

greiginsydney commented 3 years ago

Awesome! That's progress. Leave it with me.

aerialvideographyuk commented 3 years ago

Is there anything more I can do to locate the issue?

wj7106 commented 3 years ago

I also encounter this problem. After I uninstall the oray intelligent networking client, everything is normal. If I don't uninstall, I can't copy the photos to the PI, and of course I can't FTP them to the server

lines 1-10/10 (END) ● celery.service - Celery Service Loaded: loaded (/etc/systemd/system/celery.service; enabled; vendor preset: enabled) Active: failed (Result: timeout) since Tue 2021-05-11 21:38:01 CST; 57s ago Process: 396 ExecStart=/bin/sh -c ${CELERY_BIN} -A ${CELERY_APP} multi start --logfile=${CELERYD_LOG_FILE} --logl

May 11 21:36:05 raspberrypi systemd[1]: Starting Celery Service... May 11 21:38:01 raspberrypi systemd[1]: celery.service: Start operation timed out. Terminating. May 11 21:38:01 raspberrypi systemd[1]: celery.service: Control process exited, code=killed, status=15/TERM May 11 21:38:01 raspberrypi systemd[1]: celery.service: Failed with result 'timeout'. May 11 21:38:01 raspberrypi systemd[1]: Failed to start Celery Service.

aerialvideographyuk commented 3 years ago

wj7106, I haven't heard of the oray intelligent networking client - is that what you are using to be able to connect via 4g instead of using remote.it? Have you any idea why it is failing?

wj7106 commented 3 years ago

Oray intelligent network client is a virtual networking software, with 4G network, remote access without remote.it, virtual intranet access, mobile phone or PI installation common client, can access each other, but after installation, I can't copy photos to PI, also can't use FTP, I don't know where the error is, The error code of sudo systemctl status cell is the same as yours. Maybe this kind of software will occupy some system ports and other services!

greiginsydney commented 3 years ago

Hi @wj7106,

Can you please link me to this software, and perhaps the process you're following to install it on the Pi? I've been unable to dupe the issue with remote.it and it's very difficult trying to debug in this fashion.

If Oray is also manifesting the same issue, it might dupe for me and lead me to a common cause (and cure).

Thanks,

- Greig.

wj7106 commented 3 years ago

https://service.oray.com/question/5063.html https://pgy.oray.com/download/?icn=pgy_download

I chose their products. I didn't need to install any client on the system. I used their 4G router directly. After installing the client on my computer, I could build a virtual network and directly access any intranet host on the 4G router. Now there is a problem, Use raspberry zero to send back photos to my computer with vsftpd server. It takes 5 minutes for a photo with a capacity of 20m. The speed of vsftpd transmission seems to be very slow!

greiginsydney commented 3 years ago

Unfortunately those sites don't appear to translate well to English in my browser.

What were you referring to when you said "but after installation, I can't copy photos to PI", and that celery wouldn't start? It looks to me from one of those links that you've installed something - but it's a generic Linux install, not something specific to the Pi/Raspbian. At least remote.it is supported on the Pi.

aerialvideographyuk commented 3 years ago

Just for others information - I use a Huawei e8372 USB wifi 4G dongle. It plugs into a spare 5v outlet on my solar controller and gets turned on by a FET switch that uses the 5v from the spare micro sd socket on the Pi. The wifi becomes available before the Pi completes it's boot sequence. So far it seems very reliable and the transfer to dropbox is pretty quick. I can't connect to the Pi from outside it's local wifi range without remote.it and with remote.it I can connect but the copy from camera to the pi fails (celery fails to start). Would it be worth me going to version 3.0.2 - before celery was implemented and trying that with remote.it?

wj7106 commented 3 years ago

Hi, I'd like to know how many ampere hour batteries your USB 4G connects to the USB port of the solar charging controller? How does the FET switch control this? Do you have any details? thank you!

aerialvideographyuk commented 3 years ago

wj7106 - I'm happy to pass on the details but perhaps do it by mail to free this up for the issue - dave@aerialvideographyuk.com

greiginsydney commented 3 years ago

@aerialvideographyuk - can you try this please:

  1. Edit the file /etc/default/celery and change the CELERYD_LOG_LEVEL from "INFO" to "DEBUG"
  2. Reboot and confirm celery again fails to start (sudo systemctl status celery)
  3. Check if anything's logged in /var/log/celery/celery_worker.log and post it here please.

Don't forget to return logging to INFO before returning the intvlm8r into production, as the debug logging is significant.

Plan B: This shortcut might reveal something: journalctl | grep "celery"

It might be interesting to try the same with remote.it: journalctl | grep "remote.it" (or maybe just 'remote')

journalctl on its own will dump LOTS. (Press space bar to go to the next page, and Q to quit.)

Paste anything that looks interesting back here...

greiginsydney commented 3 years ago

I've been doing some more reading about journalctl.

journalctl -u celery looks to give similar output to journalctl | grep "celery".

journalctl --since "2021-05-15 14:10:10" will filter the view, as will journalctl -b, which only shows from the time of the last boot.

Hopefully something in one of those two logs will further point us to the root cause.

aerialvideographyuk commented 3 years ago

Again celery did not start and log didn’t update at all.

Here is the output from the journalctl commands

pi@AVUKcontroller1:/var/log/celery $ journalctl | grep "celery" May 18 18:17:07 AVUKcontroller1 systemd-tmpfiles[110]: [/etc/tmpfiles.d/celery.conf:1] Line references path below legacy directory /var/run/, updating /var/run/celery → /run/celery; please update the tmpfiles.d/ drop-in file accordingly. May 18 18:17:19 AVUKcontroller1 systemd-tmpfiles[203]: [/etc/tmpfiles.d/celery.conf:1] Line references path below legacy directory /var/run/, updating /var/run/celery → /run/celery; please update the tmpfiles.d/ drop-in file accordingly. May 18 18:19:05 AVUKcontroller1 systemd[1]: celery.service: Start operation timed out. Terminating. May 18 18:19:05 AVUKcontroller1 systemd[1]: celery.service: Control process exited, code=killed, status=15/TERM May 18 18:19:05 AVUKcontroller1 systemd[1]: celery.service: Failed with result 'timeout'. May 18 18:19:07 AVUKcontroller1 sudo[602]: pi : TTY=pts/0 ; PWD=/home/pi ; USER=root ; COMMAND=/usr/bin/systemctl status celery pi@AVUKcontroller1:/var/log/celery $ pi@AVUKcontroller1:/var/log/celery $ journalctl | grep "remote" May 18 18:17:22 AVUKcontroller1 systemd[1]: Started remote.it agent, manages connections. May 18 18:17:27 AVUKcontroller1 systemd[1]: Started Publish remoteit.local as alias for AVUKcontroller1.local via mdns. May 18 18:17:27 AVUKcontroller1 systemd[1]: @.: Main process exited, code=exited, status=1/FAILURE May 18 18:17:27 AVUKcontroller1 systemd[1]: @.: Failed with result 'exit-code'. May 18 18:17:32 AVUKcontroller1 systemd[1]: @.: Service RestartSec=5s expired, scheduling restart. May 18 18:17:32 AVUKcontroller1 systemd[1]: @.: Scheduled restart job, restart counter is at 1. May 18 18:17:32 AVUKcontroller1 systemd[1]: Stopped Publish remoteit.local as alias for AVUKcontroller1.local via mdns. May 18 18:17:32 AVUKcontroller1 systemd[1]: Started Publish remoteit.local as alias for AVUKcontroller1.local via mdns. May 18 18:17:33 AVUKcontroller1 bash[424]: Established under name 'remoteit.local' May 18 18:17:34 AVUKcontroller1 systemd[1]: Condition check resulted in fast remote file copy program daemon being skipped. pi@AVUKcontroller1:/var/log/celery $

All the best, Dave.

From: Greig Sheridan Sent: 15 May 2021 13:00 To: greiginsydney/Intervalometerator Cc: aerialvideographyuk; Mention Subject: Re: [greiginsydney/Intervalometerator] Images not transferring fromcamera -> Pi when using a 4G wifi dongle (#79)

@aerialvideographyuk - can you try this please:

  1. Edit the file /etc/default/celery and change the CELERYD_LOG_LEVEL from "INFO" to "DEBUG"
  2. Reboot and confirm celery again fails to start (sudo systemctl status celery)
  3. Check if anything's logged in /var/log/celery/celery_worker.log and post it here please. Don't forget to return logging to INFO before returning the intvlm8r into production, as the debug logging is significant. Plan B: This shortcut might reveal something: journalctl | grep "celery" It might be interesting to try the same with remote.it: journalctl | grep "remote.it" (or maybe just 'remote') journalctl on its own will dump LOTS. (Press space bar to go to the next page, and Q to quit.) Paste anything that looks interesting back here... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

-- This email has been checked for viruses by AVG. https://www.avg.com

greiginsydney commented 3 years ago

It's been a while but I'm back on the case today.

I've bought a Huawei E8372 dongle & Telstra SIM, and having installed remote.it, Celery no longer starts after a reboot of the Pi!!

Hopefully it will be a bit easier to debug now it's happening to me. Finally.

aerialvideographyuk commented 3 years ago

Hi Greig,

Strange isn’t it. Good luck with finding it, I’ve gone as far as my knowledge will allow and failed. As far as my problem goes it occurs even when not using the dongle ie using wifi. If there is anything I can do then give me a shout.

All the best, Dave

From: Greig Sheridan Sent: 08 June 2021 08:38 To: greiginsydney/Intervalometerator Cc: aerialvideographyuk; Mention Subject: Re: [greiginsydney/Intervalometerator] Images not transferring fromcamera -> Pi when using a 4G wifi dongle (#79)

It's been a while but I'm back on the case today. I've bought a Huawei E8372 dongle & Telstra SIM, and having installed remote.it, Celery no longer starts after a reboot of the Pi!! Hopefully it will be a bit easier to debug now it's happening to me. Finally. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

-- This email has been checked for viruses by AVG. https://www.avg.com

greiginsydney commented 3 years ago

Hi Dave.

i'd like to understand what's going on a little more, but I have a bandaid/fix: just edit /opt/remoteit-headless/scripts/remoteit-headless.service and change the "after" line to add celery, like this:

After=network.target rc-local.service celery.service

This just tells remote.it not to start until celery is up.

I'll keep tinkering with it, but I'm hoping that will resolve the issue long-term.

- G.

wj7106 commented 3 years ago

@aerialvideographyuk This solution is really quite easy to use. I don't need FTP or Internet disk upload, because what I shoot is raw format file. My country can't access Google's Internet disk or Dropbox. If I use vsftpd, the speed is too slow. The speed of FTP is really not suitable for raw format. So I go to the scene every 15 days to clean up the lens, check the battery condition and copy photos, This irf520 FET switch with Huawei 8372, remote view equipment shooting, really easy to use! thank!

aerialvideographyuk commented 3 years ago

Hi Greig,

Have now done the change and all appears to be working fine on both wifi and over 4g with the dongle. Thanks very much for your help. Once the ability to monitor the 12v battery is in place I would be happy to put it to work for real.

greiginsydney commented 3 years ago

I've added the 'fix' for this to the Troubleshooting page (although it won't be visible there until I release the next update).

Closing.